Having a hash symbol in sickchill password is causing post processing to fail.

[captainhowie]

Hi There
I have recently upgraded to Python3 and am using sabnzbd and sickchill on a QNAP NAS. I was having some trouble setting up the post processing and have discovered that the issue is with the fact that my password has a # at the end of it and this often is removed from the autoProcessMedia.cfg file after saving and running it.

Is there a restriction on the use of the # symbol in passwords for the purposes of the .cfg script and if not, is there some sort of parenthesis required to indicate that it is part of the password and not a comment.

Thanks
Andrew

[clinton-hall]

yeah a # is a "comment out character in python.... so password = test12#34 will be handled as test12
I just did a test, and if I put in quotes password = "test12#34" this handled the password as test12#34... I haven't tested as far as testing this trough the request.post (url posting)....

Also, I recommend setting up the sickchill api and using that, not the user/pass. This works better for auto-fork as well.

[captainhowie]

Thanks. I was advised to leave the API blank as there is a recent issue causing a crash during post processing - is this now resolved?

[clinton-hall]

ok, who advised? I was not aware of this issue (certainly I have not experienced it).
Try the password in quotes as above, and hopefully the api issue is resolved soon.

[captainhowie]

Sorry - I hope I'm not getting anyone in trouble here - OneCD said it was a very recent issue so leave it blank temporarily.

https://forum.qnap.com/viewtopic.php?f=320&t=132373&start=1815

[clinton-hall]

nah... not getting anyone in trouble at all... it's good when we can all share our findings/isues and try to get everything playing nice.

looks like OneCD was suggesting that the api interface causes a crash in SABnzbd postprocessing.
As I use NZBGet that may explain why I haven't seen it...

@OneCDOnly
Do you have any insights into this issue (about not using SickChill api in autoProcessMedia.cfg as this causes a SABnzbd crash)? Is there anything that I need to try and fix here or is this something to be addressed within SickChill or SABnzbd and if so has it been reported?

[captainhowie]

Thanks so much for being really prompt and responsive. Appreciate your help. Good luck with the API issue (if it is actually your issue to begin with!)

[OneCDOnly]

@OneCDOnly
Do you have any insights into this issue (about not using SickChill api in autoProcessMedia.cfg as this causes a SABnzbd crash)? Is there anything that I need to try and fix here or is this something to be addressed within SickChill or SABnzbd and if so has it been reported?

@clinton-hall sorry, I don't know a lot about this. It only happens when running the new SickChill (rewritten for Python 3), so it may be a SickChill error. @miigotu is hard-at-work re-tooling SickChill to work correctly in Python 3.

The error appears in the SABnzbd -> nzbToMedia logs as a crash. @niwa2 posted about this problem and their solution here.

Frankly, I didn't know where this issue should be reported. 🤔

Here's an example:

[22:49:02] [INFO]::MAIN: Script triggered from SABnzbd Version 3.0.0.
[22:49:02] [INFO]::MAIN: Auto-detected SECTION:SickBeard
[22:49:02] [INFO]::MAIN: Calling SickBeard:tv to post-process:City.on.a.Hill.S01E10.Mayor.Curley.and.The.Last.Hurrah.1080p.WEBRip.HEVC.x265.Rmteam
[22:49:02] [INFO]::MAIN: Attempting to auto-detect tv fork
[22:49:02] [ERROR]::MAIN: Failed to get data from JSON
Changing to directory: /share/CACHEDEV1_DATA/Download/nzbToMedia

-- Cleaning bytecode --
WARNING: Automatic cleanup could not be executed.
If errors occur, manual cleanup may be required.
REASON : Error Code: 128

Directory is not a git repository

Force cleaning folder: libs
Skipping required subfolders ['common', 'custom', 'py2', 'win']

Force cleaning folder: core
Skipping required subfolders ['auto_process', 'extractor', 'plugins', 'utils']
Returning to directory:  /share/CACHEDEV1_DATA/.qpkg/SABnzbd/SABnzbd

-- Cleanup finished --

Traceback (most recent call last):
File "/share/Download/nzbToMedia/nzbToSickBeard.py", line 273, in <module>
result = nzbToMedia.main(sys.argv, section)
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/nzbToMedia.py", line 966, in main
result = process(os.environ['SAB_COMPLETE_DIR'], input_name=os.environ['SAB_FINAL_NAME'], status=int(os.environ['SAB_PP_STATUS']),
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/nzbToMedia.py", line 853, in process
result = tv.process(section_name, input_directory, input_name, status, client_agent, download_id, input_category, failure_link)
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/auto_process/tv.py", line 56, in process
fork, fork_params = auto_fork(section, input_category)
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/forks.py", line 140, in auto_fork
rem_params, found = api_check(r, params, rem_params)
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/forks.py", line 26, in api_check
json_data = json_data['data']
KeyError: 'data'

[OneCDOnly]

[22:49:02] [INFO]::MAIN: Attempting to auto-detect tv fork
[22:49:02] [ERROR]::MAIN: Failed to get data from JSON

Just had a thought: can nzbToMedia still detect SickChill after all the recent changes? Most of these are now in the develop branch.

[clinton-hall]

Ah, I hadn't realised this was referencing the Py3 SickChill branch.

I believe this has been raised already #1759

It does appear that (at that time) SickChill was not able to parse json data due to byte vs string issues (and I think everyone who has ported something from py2 to py3 has encountered this).

So currently nzbToMedia cannot detect SickChill (py3) using autofork.

I'll see if I have time this weekend (heck, we may be back in lockdown) to try and set up SickChill Py3 and do some testing.
I still have a QNAP box, so if I use sherpa I should be able to set up this combination?

[OneCDOnly]

So currently nzbToMedia cannot detect SickChill (py3) using autofork.

It would be great to have autofork working but if not, is it possible to prevent this problem by setting a manual fork?

I still have a QNAP box, so if I use sherpa I should be able to set up this combination?

Yes, that should work fine. But, I get the same effect on my Debian 10.5 server running SickChill (develop). So, it's not really a QNAP issue. 😉

[clinton-hall]

Sorry, Realised it wasn't a QNAp issue, but I have the following options

1. Asustor Nas (currently running SickChill for my use using py2... so I don't want to break this just yet)
2. QNAP Nas (Currently not running anydownlaod Apps
3. Trying to remember how to log into my Azure VM

So I figured QNAP would be the easiest way to go.

[OneCDOnly]

2. QNAP Nas (Currently not running anydownlaod Apps

BTW: what is the firmware version on your QNAP?

[miigotu]

@clinton-hall python3 -m pip install sickchill && SickChill.py will get you a working py3 version on Linux , Windows, or mac after installing python3

If someone can get me a traceback I can fix whatever it is.

[clinton-hall]

@miigotu

AA sickchill.views.api.webapi.ApiError: param: 'subject' with given value: 'b'postprocess'' is out of allowed range '['help', 'future', 'episode', 'episode.search', 'episode.setstatus', 'episode.subtitlesearch', 'exceptions', 'history', 'history.clear', 'history.trim', 'failed', 'backlog', 'logs', 'logs.clear', 'sb', 'postprocess', 'sb.addrootdir', 'sb.checkversion', 'sb.checkscheduler', 'sb.deleterootdir', 'sb.getdefaults', 'sb.getmessages', 'sb.getrootdirs', 'sb.pausebacklog', 'sb.ping', 'sb.restart', 'sb.dailysearch', 'sb.propersearch', 'sb.subtitlesearch', 'sb.searchindexers', 'sb.searchtvdb', 'sb.searchtvrage', 'sb.setdefaults', 'sb.update', 'sb.shutdown', 'show', 'show.addexisting', 'show.addnew', 'show.cache', 'show.delete', 'show.getquality', 'show.getposter', 'show.getbanner', 'show.getnetworklogo', 'show.getfanart', 'show.pause', 'show.refresh', 'show.seasonlist', 'show.seasons', 'show.setquality', 'show.stats', 'show.update', 'shows', 'shows.stats']'
AA raise ApiError("param: '" + str(name) + "' with given value: '" + str(
AA File "C:\Users\fock_\AppData\Local\Programs\Python\Python38\lib\site-packages\sickchill\views\api\webapi.py", line 405, in _check_param_value
AA self._check_param_value(default, key, allowed_values)
AA File "C:\Users\fock_\AppData\Local\Programs\Python\Python38\lib\site-packages\sickchill\views\api\webapi.py", line 335, in check_params
AA self.subject, args = self.check_params(args, kwargs, "subject", "help", False, "string", list(function_mapper))
AA File "C:\Users\fock_\AppData\Local\Programs\Python\Python38\lib\site-packages\sickchill\views\api\webapi.py", line 637, in __init__
AA to_call = func(cur_args, cur_kwargs)
AA File "C:\Users\fock_\AppData\Local\Programs\Python\Python38\lib\site-packages\sickchill\views\api\webapi.py", line 164, in call_dispatcher
2020-08-14 20:54:39 INFO :: WEBSERVER :: Traceback (most recent call last):

so it looks like the incoming api call http://xx.xx.xx.xx:8081/api/xxxxxxxxxxxxxxxx/?cmd=help&subject=postprocess is having the param interpreted/imported as a byte code, not string and therefore not matching any available commands.

I pasted the url into the browser to test and get the same error logged in the browser

{"data": {}, "message": "param: 'subject' with given value: 'b'postprocess'' is out of allowed range '['help', 'future', 'episode', 'episode.search', 'episode.setstatus', 'episode.subtitlesearch', 'exceptions', 'history', 'history.clear', 'history.trim', 'failed', 'backlog', 'logs', 'logs.clear', 'sb', 'postprocess', 'sb.addrootdir', 'sb.checkversion', 'sb.checkscheduler', 'sb.deleterootdir', 'sb.getdefaults', 'sb.getmessages', 'sb.getrootdirs', 'sb.pausebacklog', 'sb.ping', 'sb.restart', 'sb.dailysearch', 'sb.propersearch', 'sb.subtitlesearch', 'sb.searchindexers', 'sb.searchtvdb', 'sb.searchtvrage', 'sb.setdefaults', 'sb.update', 'sb.shutdown', 'show', 'show.addexisting', 'show.addnew', 'show.cache', 'show.delete', 'show.getquality', 'show.getposter', 'show.getbanner', 'show.getnetworklogo', 'show.getfanart', 'show.pause', 'show.refresh', 'show.seasonlist', 'show.seasons', 'show.setquality', 'show.stats', 'show.update', 'shows', 'shows.stats']'", "result": "error"}

[miigotu]

I can cast that to the right type, but I'm confused why it works with a blank fork set?

[clinton-hall]

@miigotu

When you specify a fork in nzbToMedia, we just pass in all of the parameters that are defined for that fork. So we skip this step (autofork) where we actually hit the postprocess help page to get a list of possible parameters.

Then, when we actually call the postprocess command we pass all the params as a dictionary through the requests session... so it is possible that params are being correctly detected when passed as a dict, but when passed directly through the url these are causing issues.

Potentially, at this end, I could pass ?cmd=help&subject=postprocess as a dict params ={'cmd': 'help', 'subject': 'postprocess'},
and see if that works... and this would be cleaner....

However I would have expected to be able to hit the help page via url locally anyway, so possibly worth investigating that.

[miigotu]

Yeah when I responded earlier I thought that was the postprocess command (which works).

I will check the help command shortly.

[miigotu]

tornado web server for some reason explicity makes all arguments into bytes for some reason smh.
https://github.com/SickChill/SickChill/pull/6642/files#diff-1df4fcb78e368befe86dba988029d4a7R65

[miigotu]

Also, yes a dict as params will work. I could expose it for post if you want also, but for now it's only GET

[clinton-hall]

I just added the params as dict to nightly.... Can't say for sure if that really did much... but between that, and having updated SickChill to latest, the auto-fork worked.

[19:28:14] [DEBUG]::SERVER: Server responded at http://10.1.1.35:8081
[19:28:14] [INFO]::MAIN: Attempting to auto-detect tv fork
[19:28:14] [DEBUG]::MAIN: Removing excess parameters: ['delete_files', 'delete_on', 'dir', 'dirName', 'force', 'ignore_subs', 'is priority', 'proc_dir']
[19:28:14] [INFO]::MAIN: SickBeard:tv fork auto-detection successful ...
[19:28:14] [INFO]::MAIN: SickBeard:tv fork set to SickChill-api

@captainhowie
If you update SickChill and switch nzbToMedia to nightly branch, you should be able to set the apikey in autoProcessMedia.cfg and it should work correctly.

[clinton-hall]

For what it is worth... I have switched my Asustor NAS over to running SickChill develop and everything appears to be working (in terms of hand-shaking between Apps).

@miigotu I have submitted this SickChill App to Asustor App Center. It currently does a git clone and if 'SickChill.py' doesn't exist it checks out the develop branch before launching.
So in theory, once this is merged to master it will just keep users on master (default) branch. Also, since it now requires the Python3 App (not Python) it won't allow a user to upgrade without installing Python3 App first.

Behind the scenes it use venv to install all requirements into a separate environment to prevent contamination with other Apps.

[OneCDOnly]

Behind the scenes it use venv to install all requirements into a separate environment to prevent contamination with other Apps.

I'm just looking at doing the same thing with the sherpa apps. 🤓

[miigotu]

For what it is worth... I have switched my Asustor NAS over to running SickChill develop and everything appears to be working (in terms of hand-shaking between Apps).

@miigotu I have submitted this SickChill App to Asustor App Center. It currently does a git clone and if 'SickChill.py' doesn't exist it checks out the develop branch before launching.
So in theory, once this is merged to master it will just keep users on master (default) branch. Also, since it now requires the Python3 App (not Python) it won't allow a user to upgrade without installing Python3 App first.

Behind the scenes it use venv to install all requirements into a separate environment to prevent contamination with other Apps.

It is available on pypi now, and python3 comes with pip. I'm working on a pip updater in the app. Pip will be a better way than git because of git not tracking folders and people will get compiled requirements specifically for their device. I currently patch sys.path if it is not installed in site-packages or user-site, but we really want to avoid the bundled libs if possible.

[OneCDOnly]

It is available on pypi now, and python3 comes with pip. I'm working on a pip updater in the app. Pip will be a better way than git because of git not tracking folders and people will get compiled requirements specifically for their device. I currently patch sys.path if it is not installed in site-packages or user-site, but we really want to avoid the bundled libs if possible.

Wow, I just tried the pip install. Had a working SickChill in under 30 seconds! Good work @miigotu 👍🏽

[clinton-hall]

I'll have a play and see if I can package it to install via pip (as I already do for the requirements).
I was going to do this originally, but decided git was safer as I could test for the existence of SickChill.py.... but now I realise that this was flawed thinking... pip in python 3 will only install the python 3 version anyway!

With the App I created, the one thing that didn't appear to work was the updater from within SickChill. But the App was always set up to update on each restart anyway...

This new App is much cleaner than the previous versions. And by going to pip I think this will really be quite clean...

@OneCDOnly - I'll try and send through the code I am using to set these up (so far I have used this methodology for SABnzbd and SickChill)... given that the Apps I maintain for Asustor used the same methodology as the Apps I maintained for QNAP, it may be interesting to compare. FWIW, the pip method in Python3 (in my opinion) is so much nicer than having to try and export lib paths for Optware/Entware or having to cross compile for every architecture and bundle!!!

[OneCDOnly]

FWIW, the pip method in Python3 (in my opinion) is so much nicer than having to try and export lib paths for Optware/Entware or having to cross compile for every architecture and bundle!!!

Definitely! The way of the future! 😄

I think this is how I'll rejig the QPKGs moving-forward. Just need to ensure Python3 is there (via Entware because QNAP's version is awful), then install everything via pip.

[miigotu]

I'll have a play and see if I can package it to install via pip (as I already do for the requirements).
I was going to do this originally, but decided git was safer as I could test for the existence of SickChill.py.... but now I realise that this was flawed thinking... pip in python 3 will only install the python 3 version anyway!

With the App I created, the one thing that didn't appear to work was the updater from within SickChill. But the App was always set up to update on each restart anyway...

This new App is much cleaner than the previous versions. And by going to pip I think this will really be quite clean...

@OneCDOnly - I'll try and send through the code I am using to set these up (so far I have used this methodology for SABnzbd and SickChill)... given that the Apps I maintain for Asustor used the same methodology as the Apps I maintained for QNAP, it may be interesting to compare. FWIW, the pip method in Python3 (in my opinion) is so much nicer than having to try and export lib paths for Optware/Entware or having to cross compile for every architecture and bundle!!!

You don't need to worry about requirements, they come with pip install sickchill (I still recommend a venv or Virtualenv though).
You also don't have to try and figure out the path to git or have an app directory set up. Just the service files to make it run automatically and any related images for their service manager UI.

Idk if asustor or qnap has a $HOME set, but default config/db path is also now $HOME/sickchill

[OneCDOnly]

Idk if asustor or qnap has a $HOME set, but default config/db path is also now $HOME/sickchill

In QNAP, everything runs as 'admin' (root user). Unfortunately, the home for admin is /root which is a tmpfs (non-persistent) and is quite small. Even have to be careful with pip installs as the default pip cache is in /root (which is also /). So, after about a dozen packages are downloaded, you've got a full drive, along with a chain of errors. I've learned the hard-way to specify a different cache location when installing lots of Python modules.

That's OK, the reason QPKGs exist is to construct an environment around the needs of the app. More work for me to do. 😆

[clinton-hall]

so 2 things my App does...

1. As part of post install (so only runs the once) is to make sure we have latest pip, and venv and then create the virtual environment inside of the App Directory

               cd $PKG_BASE_DIR
		python3 -m pip install --upgrade pip
		python3 -m pip install --user virtualenv
		python3 -m venv env

2. Then within the start.stop script for the App itself

DAEMON="python3 SickChill.py"
DAEMON_OPTS=" --daemon --datadir $PKG_DIR/config --pidfile $PID_FILE"
........

	cd $PKG_DIR
	#set venv
	source env/bin/activate
	python3 -m pip install sickchill #this should just skip on subsequent starts... this could be put into the post install script instead?
	$DAEMON $DAEMON_OPTS

[clinton-hall]

Yes, by default the code above will execute as root in Asustor... but by using the venv it isolates where the modules are installed.
Also, all App versions have specified the --datadir and this directory is built in to the App to be copied over on each upgrade of the App itself (but if you remove the App, you do loose the data).

What I haven't shown there, as more specific to the start-stop daemon in Asustor, is I set all to run as admin user (not root)

PKG_NAME="SickChill"
PKG_DIR="/usr/local/AppCentral/sickbeard-tvrage"
RUN_AS="admin:administrators"
CONFIG_DIR="$PKG_DIR/config"
PID_FILE="$PKG_DIR/$PKG_NAME.pid"
DAEMON="python3 SickChill.py"
DAEMON_OPTS=" --daemon --datadir $PKG_DIR/config --pidfile $PID_FILE"

..........
	/sbin/start-stop-daemon -S -n $PKG_NAME -p $PID_FILE -c $RUN_AS -x ${DAEMON} --${DAEMON_OPTS}

Note, the original package path is also sickbeard-tvrage because of the history of the App. If I change that, it would break the history and require everyone to install new!

[miigotu]

Can we please make the app folder and data folder "sickchill", I have renamed every reference to sickrage/tvrage and even SickBeard is being refactored into the sickchill package. This also avoids future conflicts, because my db is going to be changing a lot soon.

[miigotu]

Totally recommend everyone installing a new package. Maybe give an update to the old package that does it for them or directs them to do it. Literally every git user is going to have to do something manual to be able to use the python3 branch (by design)

[captainhowie]

Sorry for the late reply to all of this. Thanks for resolving the issue although I have no idea which branch to install (or how to change branches).
All of the apps (Nzbtomedia, sickchill and sabnzbd) have been installed via OneCD Sherpa.

Any hints welcome.
Thanks for all the work guys

[miigotu]

yeah a # is a "comment out character in python.... so password = test12#34 will be handled as test12
I just did a test, and if I put in quotes password = "test12#34" this handled the password as test12#34... I haven't tested as far as testing this trough the request.post (url posting)....

Also, I recommend setting up the sickchill api and using that, not the user/pass. This works better for auto-fork as well.

@captainhowie your question was answered in the second post. Put quotes around it.

[captainhowie]

yeah a # is a "comment out character in python.... so password = test12#34 will be handled as test12
I just did a test, and if I put in quotes password = "test12#34" this handled the password as test12#34... I haven't tested as far as testing this trough the request.post (url posting)....
Also, I recommend setting up the sickchill api and using that, not the user/pass. This works better for auto-fork as well.

@captainhowie your question was answered in the second post. Put quotes around it.

Thanks @miigotu Perhaps I should have been clearer - the discussion has proceeded much further beyond the initial query onto the error in relation to the API link and @clinton-hall suggested I change nzbmedia to a different branch. Thanks

[captainhowie]

I just added the params as dict to nightly.... Can't say for sure if that really did much... but between that, and having updated SickChill to latest, the auto-fork worked.

[19:28:14] [DEBUG]::SERVER: Server responded at http://10.1.1.35:8081
[19:28:14] [INFO]::MAIN: Attempting to auto-detect tv fork
[19:28:14] [DEBUG]::MAIN: Removing excess parameters: ['delete_files', 'delete_on', 'dir', 'dirName', 'force', 'ignore_subs', 'is priority', 'proc_dir']
[19:28:14] [INFO]::MAIN: SickBeard:tv fork auto-detection successful ...
[19:28:14] [INFO]::MAIN: SickBeard:tv fork set to SickChill-api

@captainhowie
If you update SickChill and switch nzbToMedia to nightly branch, you should be able to set the apikey in autoProcessMedia.cfg and it should work correctly.

@miigotu to save you trawling through the history

[miigotu]

https://github.com/clinton-hall/nzbToMedia/wiki/Branches#switching-branches-in-nzbtomedia

[miigotu]

@clinton-hall I see sickchill is missing from your wiki xD

[clinton-hall]

@miigotu
To be honest, it was set to be community editable, so I'll check to be sure it wasn't added and then removed... but generally I referenced sick* forks and the people specifically added some of the various forks.

I will ensure SickChill has a dedicated section on here.

[miigotu]

@clinton-hall SickChill is gonna need its own section on the wiki soon lol
https://github.com/SickChill/SickChill/compare/a353182993718fc6ee117e0edcb54e8b2cecb57d..aebe5c82ed3ced25397c702f6cc39449b549892e

[clinton-hall]

I have added SickChill to the home page, and linked that directly to the nzbToMedia config details on your wiki.

I also sorted all (known) supported TV management systems in Alphabetical order (to try and avoid any issues there)... I don't expect this page to be a "sales" point. I have tried to keep neutral in most of this, and simply link to the projects that are supported.

On that. I do suspect that SickChill and a few other projects will soon become quite far removed from SickBeard that I will eventually need to move to separate config sections and handling....
I have been trying to avoid having to make users move all of their config etc (particularly where the config is actually imported in from NZBGet), so I have been trying to keep them all handled under the same SickBeard section.

Thinking about it, I could use the current auto-fork detection to write all config over to a specific SickChill (or other) section etc... But the problem I still face is with the config imported from NZBget. I could just introduce the new config details, and continue to run on the previous config until such time and new values were sent through from NZBget.... It's just going to take some time and testing (multiple branches, some of which I don't have set up!).

[clinton-hall]

@miigotu
Sorry, I just checked your diff.... now I get it!

Yes... this would justify having your own section....

I recall when the conversation existed about adding TV series to CouchPotato.... Alas, I will be sad to stop using my CouchPotato, as I did contribute a lot of time to supporting that project.... but I will be keen to test SickChill on my movie collection 👍

[captainhowie]

https://github.com/clinton-hall/nzbToMedia/wiki/Branches#switching-branches-in-nzbtomedia

Thanks @miigotu I get this error when trying to run the checkout nightly?

[/share/.qpkg/nzbToMedia/nzbToMedia] # git checkout nightly
error: pathspec 'nightly' did not match any file(s) known to git

[clinton-hall]

@OneCDOnly
How can a user change branches for nzbToMedia when installed from Sherpa?

[miigotu]

@captainhowie I imagine first you need to git fetch origin

[OneCDOnly]

@OneCDOnly
How can a user change branches for nzbToMedia when installed from Sherpa?

@clinton-hall
This line would need to be changed in the service script:

readonly SOURCE_GIT_BRANCH=master

... then the package would need to be cleaned:

/etc/init.d/nzbToMedia.sh clean