nzbToMedia error - No module named imghdr

[captainhowie]

I have just noticed a new error in sabnzbd regarding the post processing. The error is as follows:

Traceback (most recent call last):
File "/share/Download/nzbToMedia/nzbToMedia.py", line 730, in
import core
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/nzbToMedia/core/init.py", line 52, in
from core import logger, main_db, version_check, databases, transcoder
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/nzbToMedia/core/databases.py", line 11, in
from core.utils import backup_versioned_file
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/nzbToMedia/core/utils/init.py", line 13, in
from core.utils.common import clean_dir, flatten, get_dirs, process_dir
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/nzbToMedia/core/utils/common.py", line 14, in
from core.utils.files import list_media_files, move_file
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/nzbToMedia/core/utils/files.py", line 14, in
import beets.mediafile
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/nzbToMedia/libs/common/beets/mediafile.py", line 51, in
import imghdr
ImportError: No module named imghdr

Any ideas what this issue relates to?
Running on a QNAP NAS.

Many thanks
Andrew

[clinton-hall]

#1758
#1762

Basically the Python QPKG is missing standard python modules.
My recommendation is to move over to Sherpa qpkg and install SAB via that... all works well.

[captainhowie]

Thanks @clinton-hall. I am already running the Sherpa qpkg. I recently updated both SAB and sickchill to the latest via Sherpa and have since had the above error. I'll update again and see how it goes

[clinton-hall]

@OneCDOnly
Any idea why @captainhowie would get this on Sherpa? Does this still pick up python from another package?

[OneCDOnly]

It looks like it's found one of QNAP's Python's again - probably the internal Python 2.7.5. 😞

I may be able to fix this by including the same $PATH and $PYTHONPATH exports I used in the other sherpa apps in the nzbToMedia QPKG.

Working on this now...

[OneCDOnly]

OK, just pushed a new nzbToMedia QPKG (200831). It now exports $PYTHONPATH like the other Python apps.

However, I'm not sure this will fix it as I'm not exactly a Python guru.

@captainhowie can you please check which Python is run? Your NAS should show something like mine:

[~] # type -a python
python is /opt/bin/python
python is /usr/local/bin/python

[~] # ll /opt/bin/python
lrwxrwxrwx    1 admin    administ         9 Aug 30 06:33 /opt/bin/python -> python3.8*

[clinton-hall]

Interesting I don't think pythonpath will have any impact (and happy to be proven wrong)... This will add the location for Python modules, but only for the nzbToMedia install which doesn't call Python, and this won't change which python interpreter is called...

Since the script is called from SAB, I believe it is the SAB package that is finding the "other" Python in the PATH. (however SAB still starts fine since you define readonly PYTHON=/opt/bin/python3 and then call readonly LAUNCHER="cd $QPKG_REPO_PATH; $PYTHON $TARGET_SCRIPT_PATHFILE --daemon --browser 0 --config-file $QPKG_INI_PATHFILE --pidfile $DAEMON_PID_PATHFILE" to lauch the process with $PYTHON

I believe the easiest way to change this would be to either:

1. add export PATH = /opt/bin:$PATH to sabnzbd3.sh... This will ensure that all /opt/bin binaries are picked up by the child processes... but may also have unwanted consequences.
2. Set Python(3) venv, set the source and use this to run SAB... This will automatically add the virtual Python location to the start of PATH, and ensure that all SAB processes and child processes will call that python version. (we discussed this in another thread Having a hash symbol in sickchill password is causing post processing to fail. #1765 (comment), and on Asustor NAS this solution resolved similar issues)

[OneCDOnly]

add export PATH = /opt/bin:$PATH to sabnzbd3.sh... This will ensure that all /opt/bin binaries are picked up by the child processes... but may also have unwanted consequences.

Which we can do, bit I think it will mess-up the next git pull. I've previously had errors like "unstaged changes" or similar when I edited the files that were to be updated.

Set Python(3) venv, set the source and use this to run SAB... This will automatically add the virtual Python location to the start of PATH, and ensure that all SAB processes and child processes will call that python version. (we discussed this in another thread #1765 (comment), and on Asustor NAS this solution resolved similar issues)

I still haven't resolved the problems with this method. It seems that if we install all the pip modules required by each app (in requirements.txt), this causes conflicts with the OpenWRT Python 3 modules.

I need to work out how to keep the OpenWRT modules as the first preference, but this is outside my area of knowledge, so I've set it aside for now.

[OneCDOnly]

add export PATH = /opt/bin:$PATH to sabnzbd3.sh... This will ensure that all /opt/bin binaries are picked up by the child processes... but may also have unwanted consequences.

Which we can do, bit I think it will mess-up the next git pull. I've previously had errors like "unstaged changes" or similar when I edited the files that were to be updated.

Oh wait, I misread your suggestion.

Yes, we already export in sabnzbd3.sh: https://github.com/OneCDOnly/sherpa/blob/9b5058072aa20a2232ed748f9416f3228d25b666/QPKGs/SABnzbd/shared/sabnzbd3.sh#L66

... although, the way I wrote the logic for it, it only gets prepended to $PATH if it doesn't already exist anywhere in $PATH. I should change that to an automatic remove and prepend.

[clinton-hall]

Ah.. I see... yes, make /opt/bin always sit at front of PATH...

As far as the pip issues, These should not interfere with OpenWRT if it is done within venv... as these are only installed for, and called by, each App with its own separate environment.

I might see if I get Sherpa going on my QNAP and test... I'll let you know what I find.

[captainhowie]

@captainhowie can you please check which Python is run? Your NAS should show something like mine:

[~] # type -a python
python is /opt/bin/python
python is /usr/local/bin/python

[~] # ll /opt/bin/python
lrwxrwxrwx    1 admin    administ         9 Aug 30 06:33 /opt/bin/python -> python3.8*

First part is identical second part

lrwxrwxrwx 1 admin administrators 16 2020-08-13 18:09 /opt/bin/python -> /opt/bin/python3*

[captainhowie]

Hi everyone. Apologies I didn't get a chance to update the forum. I updated all three packages via sherpa yesterday morning prior to the conversation above (sabnzbd, nzbtomedia, sickchill). I ran a tv program through last night and it successfully post processed. It appears something updated between the update a few days ago, and the update available 36hrs ago. Thanks team

[OneCDOnly]

Ah.. I see... yes, make /opt/bin always sit at front of PATH...

Done: OneCDOnly/sherpa@b8dbebb

I’ll mod the other apps shortly. 🤓

[clinton-hall]

@OneCDOnly I have a working version of sab using venv... I sent you a PR (you don't have to merge this, as it does undo changes you are recently made) but I figured this may be helpful (seeing as Entware introduced some fun challenges).

[clinton-hall]

and a PR for SickChill.... test and use as you will.... Happy to answer any questions about it.

[OneCDOnly]

@clinton-hall thank you. Glad to see it can be done via pip - I guessed it would take someone who knows Python better than I. 😄

I hope to revisit the pip installs again soon, but I'm trying to get sherpa refactored to handle multiple package operations. So, I really need to stop playing with the service scripts for a while. Hopefully, they're stable enough now to be left alone.

I'll work on adding those mods when I can. Cheers!

[clinton-hall]

All good.... I'll leave my fork there with the PRs so you can review when you are ready to look at this... I personally found this cleaned up my apkgs so much and has made the packages easier to maintain.

FWIW, I think the entire

        cd $QPKG_PATH
	$PYTHON -m pip install --upgrade pip
	$PYTHON -m pip install --user virtualenv
        # create venv without pip due to entware pip issues.
	$PYTHON -m venv --without-pip env
        # activate venev
        source env/bin/activate
        # Within it, invoke this well-known script to manually install pip(1) into env
        curl https://bootstrap.pypa.io/get-pip.py | python3
        deactivate
        # Reactivate now including the pip(1) command.
        source env/bin/activate
        # And finally, upgrade pip(1) itself  
        python3 -m pip install --upgrade pip
        # now install sickchill
        python3 -m pip install sickchill
        deactivate

block would be better off sitting in the installation routine and only done/checked once.
Then in the main service start script just call

    cd $QPKG_PATH
    source env/bin/activate
    # check for update... this may be an optional step?
    python3 -m pip install --upgrade sickchill

then start the daemon with appropriate arguments. In the case of SickChill, just execute SickChill.py (with arguments to be passed) since this is in the PATH within the venv.
Likewise you'll note that when in the venv, you just call either python or python3 from the PATH and DO NOT call /opt/bin/python3 as this is outside of the venv (I got that wrong a few times).

There is probably a lot more cleaning up that can be done too. i.e. remove the git checks (Still required for SAB but not SickChill).