TorrentToMedia doesn't work from qBt

[DecipherIt]

Describe the bug
I started setting up everything again (long time user) but for some reason TorrentToMedia doesn't launch from qBt when a download is complete/ No logs, nothing shows up.
It runs just fine manually.

Technical Specs

1. Win 10 x64 '....'
2. Python 3.8.'....'
3. qBittorrent '....'
4. SickGear (SickChill, CouchPotoato, Radarr, Sonarr) '....'

Expected behavior
I expect the download to finish and then TTM would kick in like it always did but it doesn't on my new setup.
On Completion string is: D:\Users\Computer\AppData\Local\Programs\Python\Python38.exe D:\nzbToMedia\TorrentToMedia.py "%D" "%N" "%L" "%I"

Log
No logs available since it doesn't start.

[clinton-hall]

Any logs in qBT?

[DecipherIt]

I don't seem to be able to locate any. There are none in the qBt dir.

[DecipherIt]

I really have no idea how to fix this. The SG bat works fine but I don't want to xcopy

[clinton-hall]

When you say it runs ok manually. How are you running it? using D:\Users\Computer\AppData\Local\Programs\Python\Python38.exe D:\nzbToMedia\TorrentToMedia.py

[DecipherIt]

Just found the log dir it says nothing much.
It starts in paused, I manually start it.

(N) 2020-09-24T20:28:45 - Download first and last piece first: Off, torrent: 'MyTest.2020.S01E09.720p.HMAX.WEB-DL.DD5.1.H.264.mkv'
(N) 2020-09-24T20:28:45 - 'MyTest.2020.S01E09.720p.HMAX.WEB-DL.DD5.1.H.264.mkv' added to download list.

Then it does nothing on fhinish.

With this exact string
D:\Users\Computer\AppData\Local\Programs\Python\Python38.exe D:\nzbToMedia\TorrentToMedia.py "%D" "%N" "%L" "

Running the command (which had an error) gives me this

D:\Users\Computer\AppData\Local\Programs\Python\Python38\Python38.exe D:\NzbToMedia\nzbToMedia\TorrentToMedia.py
'D:\Users\Computer\AppData\Local\Programs\Python\Python38\Python38.exe' is not recognized as an internal or external command,

Python works fine with Radarr.

[DecipherIt]

Command is
D:\Users\Computer\AppData\Local\Programs\Python\Python38.exe D:\NzbToMedia\nzbToMedia\TorrentToMedia.py "%D" "%N" "%L" "

Still the above error, python is added to enviromental.

[clinton-hall]

D:\Users\Computer\AppData\Local\Programs\Python\Python38\Python38.exe is not recognized as an internal or external command

Check this path... if it doesn't exist, then calling this will not work.

For me the path is: C:\Program Files\Python37\python.exe
If yours is installed into D:\Users\Computer\AppData\Local\Programs I would still expect this to be D:\Users\Computer\AppData\Local\Programs\Python38\Python.exe or similar...
Find out what the actual location of your python.exe is, then check that the script runs. Then put that path into qBt.

[DecipherIt]

D:\Users\Computer\AppData\Local\Programs\Python\Python38 this path exists.
Python38.exe is in it.
I could try and run Python.exe instead of python38.exe.

[DecipherIt]

Running python.exe on the path instead of python38.exe does run in CMD.

[DecipherIt]

now it runs but failes

https://gist.github.com/DecipherIt/3b6f61eacc1c4236cefe7ebbeee594c4

[DecipherIt]

fixed permissions.

Now i'm still getting
[00:14:28] [POSTPROCESS]::SICKBEARD: SUCCESS: The download succeeded, sending a post-process request
[00:14:28] [DEBUG]::SICKBEARD: Opening URL: http://localhost:43728/home/postprocess/processEpisode with params: {'dir': 'M:\ProcessTV\tv\test.2020.S01E04.test.720p.HMAX.WEB-DL.DD5.1.H.264-test', 'failed': 0, 'quiet': 1, 'nzbName': 'test.2020.S01E04.test.720p.HMAX.WEB-DL.DD5.1.H.264-test'}
[00:14:28] [ERROR]::SICKBEARD: Server returned status 401

[DecipherIt]

I want it to symlink (if possible with vero/pi3) it looks like it did drag a few files to the process dir but failes afterwards (sorry the edit button doesn't work for some reason).

[DecipherIt]

I want to symlink because it moves between drives that aren't in RAID setup (if that's possible) on a win 10 pc.

Here is a gist of my sickbeard part of the config (using sickgear), 401 is unauthorized but I have a username but no password.

https://gist.github.com/DecipherIt/038de060f697ff8686642295dcb277e8

[clinton-hall]

ok, if you have fixed the permissions, which now allow the script to do the hard/sym linking, then that should be fine. The issue now appears with the way the script tries to call SickGear.

I recommend setting this up to use api (not user pass... and if you currently don't use user/pass, then SickGear has disabled this!)
and also use auto fork.

so in autoProcessMedia.cfg

[SickBeard]
    [[tv]]
        enabled = 1
        host = localhost
        port = 44444
        apikey =<enter your sickgear api key>
        username =
        password =
        web_root =
        ssl = 0
        fork = auto

[clinton-hall]

Just saw your last post....

For security reasons SickGear disabled the use of no pass.
Basically someone had found a way to access someone's Sick* installation (can't recall, but don't think it was SickGear) and use the webui to access the postprocessing scripts and installed a script that then downloaded some malicious content.
The nature of the code was targeted towards many SickBeard forks, including SickGear, so once notified, most of the SickBeard forks implemented changes that prevented external access without user and pass.

I'm not 100% sure how this is implemented, but in any case I have 2 recommendations:

1. Use Api key and Auto fork as per suggestion above
2. Secure your webUI in SickGear (and indeed in all Apps) using a password

If you don't do suggestion 2. and it is just a suggestion. Please check that you don't have port 44444 externally available (i.e. turn off port forwarding in your router)

[clinton-hall]

#1721

This is the detail regarding the exploitation of a user with User and No Pass protection...

[DecipherIt]

Where can I find the API? The username/pass is more of a formality anyway, the device is encrypted so the script doesn't have to be and the connections are firewalled so they are only for allowed clients.

[DecipherIt]

Ok nvm I found it.

So how would I set up sickgear to make it symlink properly so it works with Vero's and other OSMC devices? (I know that's not your area but I'm sure you know a little haha) thank you for your help so far.

[DecipherIt]

Weird, every user has full permission now but still this in SG

01:47:34 INFO API^WEB_1 :: Folder name (Show.S03E04.test.Unmasked.720p.AMZN.WEB-DL.DD+5.1.H.264-) appears to be a valid release name. Using it.
01:47:34 ERROR SG.HELPER :: Failed creating T:\Shows\Show\Season 03 : [WinError 5] Access is denied: 'T:\\Shows\\Marvels Spider-Man\\Season 03'
01:47:34 WARNING API^WEB_1 :: Did not use file M:\ProcessTV\tv\Show.S03E04.Test.Unmasked.720p.AMZN.WEB-DL.DD+5.1.H.264-\Show.S03E04.Test.Unmasked.720p.AMZN.WEB-DL.DD+5.1.H.264-.mkv Post Processing Failed

[DecipherIt]

Still keep getting permission errors god I don't know how to fix this, everything is set to Everyone "modify" and owned my my username....

[DecipherIt]

https://gist.github.com/DecipherIt/5ecd45d3303a302e770379b1bc085e1b full gist

[clinton-hall]

So what I can see here is that nzbToMedia is failing to Symlink and SickGear is failing to create the Show's Season 03 directory...

So there is certainly something going on with permissions there.
What are your M: and T: ? are they physical disks on your PC, or are these network mounts?

[DecipherIt]

They are physical disks (mounted with Veracrypt, read-only disabled) I set the disks to be owned by the user already and allowed modify for any user but idk what's going wrong. This never happened on my previous setup (computer died, cloned disk, restored).

[DecipherIt]

Or no you mean physical disks as inside the PC, no they're removable disks (external disks)

[DecipherIt]

I have set all drives to allow every user any access but I still get the same issue on win 10, I never had this before and radarr is able to write to it so I have no idea what's going on...

[clinton-hall]

Radarr uses different language, so that is not surprising.
Both SickGear and nzbToMedia use python. So it is likely python that doesn't have permissions to write to those drives.

I do note your python location looks as though it may have been installed for a single user only? If so, try installing for all users (out side of a user's app location) to see if that provides more privillege to python.

[DecipherIt]

It's installed for all users. I had issues with radarr at first as well but just taking over the drive (owner of the drive changed to primary user instead of Admin) worked for that.

I have an SSD as C: drive and all installs go into D: (7200rpm drive) to save on wearing the SSD.

Do you have any idea or suggestions on how to fix this? I was able to run the .bat file from SG as well without any issues but I don't want to xcopy my stuff!

[DecipherIt]

When I check Windows Key + R, type netplwiz my user account is Admin as well, I seriously have no idea what's going on! I really need to get this fixed because this is a huge pain in the...

[clinton-hall]

I'm sorry, but I really don't know what to suggest.

Both SickGear and nzbToMedia use python's os module. This is failing to create links and directories.

you could try some tests on other (non encrypted) drives, you could try running the script as administrator.

basically, if you run python then

import os
os.makedirs("D:\\Users\\Computer\\test")

if this fails then something is wrong with python
if it works, then (firstly delete the new folder D:\Users\Computer\test) try the following

import os
os.makedirs("T:\\Shows\\test")

If that fails, then the issue is with python accessing T:
if that works, again delete the new test folder, then the issue is not with python, and it may have been a conflict with existing directories when SickGear tried to process.

If all of this is working, then the next step will be to test symlinkng in python

[clinton-hall]

in netplwiz, what Group(s) is your user in?

[clinton-hall]

You cannot hard-link a directory.
Sym-link, yes. It is just a shortcut. But hard-links are new file headers pointing to the same data. By definition they only exist for files.

When attempting to hard-link a directory, what should actually happen is create the new directory, then hard-link each file from the existing directory into the new directory.

Not sure what is going on with the deleted download folder. In all the nzbToMedia logs I didn't see any deletion happening in the download folder (only the Processed folder).

[DecipherIt]

I always hard linked directories, even with TTM set on hard haha. That was my default setting for years.
Now I just hard-link clone in shell extension, it seems to work manually but idk how it works automatically.

So how should I continue, I set my nzb config to hard instead of sym, it's more work for me but it seems to work. With some exceptions (please se above).

I know! It's really odd. I even did a chckdsk, nothing weird happening but I'm recording like 20 things again just to fill the directories.
This could be the issue that screwed up sym link as well.
Hardlink
filefilefile <---- source ---> filefilefile
delete one and it's source --> filefilefile

Soft links are
Link<--- source
Link<--- no source and it breaks.

That could be the only thing I could think of. But it's still really odd.

[DecipherIt]

This is an excellent example of TTM stopping to work.
It runs, links the files, then calls SG, but doesn't get a response instantly (I would assume) and then kills the process without any errors.

https://gist.github.com/DecipherIt/5a8492e26017977c1d1576c776193681#gistcomment-3484247

If this a time-out issue it should recheck the connection and verify or something... all files get left in the Post Processing dir, I just need to click process manually in SG and it all works so TTM failed in this situation.

[DecipherIt]

It also seems to manually trigger a full manual scan when files are rechecked (I had to record a few things again so they where rechecking and recording again), it suddenly started to do a full manual scan of the directory, moving files still being recorded an partially moving them.
Something is going really wrong there because I did not execute them and TTM should not go to manual mode when a file is called (even if it fails) in qBt.

[DecipherIt]

So these are the 2 main issues now..
TTM suddenly stopping, (maybe because SG doesn't respond to the api right away?) but then closing without error or retry, it also leaves the file on stopped in my qBt client (if possible I would like to turn that off, it doesn't have put downloads on stop when post processing because it hardlinks).

And it triggers manual full scans (on fail I can only assume), which isn't a huge problam normally but if things are still running it starts processing recording data, which doesn't work well.

[DecipherIt]

I just noticed that TTM often gets stuck in qBt, piling up processes. A file finishes, opens TTM but then gets stuck. I do not see ay reason why it does. There are multiple processes running under qBt and when I try to restart (throu ProcessExplorer, restart) I get a ful refresh instead of just the file.

[DecipherIt]

It seems like sometimes a process completes, qBt gets triggered and then moves into manual mode instead of selected mode.

As you can see below, TTM got triggered before and processed a file, then another file finished and it went into manual mode.

(third box)

https://gist.github.com/DecipherIt/5a8492e26017977c1d1576c776193681#gistcomment-3484247

[DecipherIt]

And again a file finished and it starts to do a full scan of the dir in SG.

[clinton-hall]

Not sure why the script is failing. Without seeing any logging of the failure, but I am sure it is not related to timeout issues, since timeout is handled within the script.
The only way I can see this happening is if SG responds with an incompatible response code.... The only theoretical issue that could come here is if r.json()['result'] doesn't exist...
I can try adding a catch here.... but in theory, since we already check for a valid response from SG, I don't know how this doesn't exist....

Also, if the script fails, I don't understand why it is "hung" in qBT...

TTM cannot start a full scan unless it is called manually without the right number of arguments. I suspect this might come when you do the restart... i.e. this doesn't pass the correct parameters.

[DecipherIt]

I will add logs of something that I know finished but didn't happen soon. Some things finish and it works fine and others finish and they don't get added (same show, same source, same destination, 7 seasons, 1 season hardlinked and processed, 6 in client still active).

Stuck status is weird to describe, as you can see above in the gist, it no longer recognizes it as a running process (closed) but doesn't start the file again or finish the process. Then it simply closes down, doesn not mark it as resume but leaves it on stop and starts a new process without me manually doing anything.

Something that seems to be even worse. Are 2 things.

I set my conf to hardlink (i can copy paste cfg if you want to see that as well) but when I restart qBt (latest 4.2.5) some files are simply gone and have to be recorded again (I checked the dirs, they're all empty). So it moves them instead.This causes a lot of issues because some things are 40 gb or bigger, right now I have a requeue of nearly 200GB which takes a days to finish again and I only have so much time for testing it.

And the other big issue is that every single time something that doesn't have a label. or the wrong label (say books), gets triggered it starts the manual post processing process and triggers SG for every file already processed (not properly removed or linked manually for testing purposes). Which causes a lot of issues I never had in the past using the D:\Users\Computer\AppData\Local\Programs\Python\Python38\python.exe D:\NzbToMedia\nzbToMedia\TorrentToMedia.py ""%D"|"%N"|"%L"|"%I""` string in uTorrrent (now using the same in qBt).

I remember running TTM as a process without any issues, ever, now that I migrated, even with the hardlink (same setup as before with the only change being client qBt) i've been having nothing but problems that could lead to inactivated accounts if I don't monitor it manually and frequently so that's a pretty big problem for me.

Tell me if you need to see anything and I'll do my best to show you as much of the required logs. I just want this up and running. I had TTM running for years without ever having to look at it.

[DecipherIt]

Also for completion, I did a chkdsk of the disk and a full cluster check, no errors. Brand new drive, no notices in S.M.A.R.T. of any failure so data did not get corrupt or anything similar either.

[DecipherIt]

failure in manual run (no failure noticed, it just moves on)

https://gist.github.com/DecipherIt/ddd6b11b6643e3dfb7dc6cfeba38c63f

[DecipherIt]

It also shows the on completion log of both SG and NZB where NZB says everything worked and SG says failed.

[DecipherIt]

Could you maybe code in a off/on swtich for manual mode (on by default or off by default), or if the script gets started with arguments that it won't switch to manual mode.
It might just be manual mode that causes half of my problems.

[clinton-hall]

have added a bypass for manual execution in latest nightly branch.
Update to latest nightly then add/set the following config items

[nzb]
    no_manual = 1

[torrent]
    no_manual = 1

[DecipherIt]

How can I do a quick commit without running manual mode?

And where should I add this in the conf?

[DecipherIt]

also did you read all the gists?

[DecipherIt]

I added them as such

[Nzb]
    no_manual = 1
	###### clientAgent - Supported clients: sabnzbd, nzbget
    clientAgent = sabnzbd
    ###### SabNZBD (You must edit this if you're using nzbToMedia.py with SabNZBD)
    sabnzbd_host = http://localhost
    sabnzbd_port = 8080
    sabnzbd_apikey = ""
    ###### Enter the default path to your default download directory (non-category downloads). this directory is protected by safe_mode.
    default_downloadDirectory = ""

[Torrent]
    no_manual = 1
	###### clientAgent - Supported clients: utorrent, transmission, deluge, rtorrent, vuze, qbittorrent, synods, other
    clientAgent = qbittorrent

[DecipherIt]

but does not work

[DecipherIt]

it removes the lines when updating.

Please give me the version I should use and how to check it out..

[DecipherIt]

Could you please help me with this checkout @clinton-hall it might solveabout 70% of the problems.

[clinton-hall]

cd D:\NzbToMedia\nzbToMedia
git checkout nightly
git pull

then re-add the 2 parameters as you already had done.

[DecipherIt]

D:\NzbToMedia\nzbToMedia>git checkout nightly
Switched to a new branch 'nightly'
Branch 'nightly' set up to track remote branch 'nightly' from 'origin'.

D:\NzbToMedia\nzbToMedia>git pull
Already up to date.

Well that's odd... I didn't update it yet.

[DecipherIt]

And now it stopped working all together. File finishes, does not start TTM anymore... TTM log shows nothing.

qBt string is still D:\Users\Computer\AppData\Local\Programs\Python\Python38.exe D:\nzbToMedia\TorrentToMedia.py "%D" "%N" "%L" "%I"

[DecipherIt]

qbt passes it, but nothing here is the log

https://gist.github.com/DecipherIt/ac7c5f986e82f439c4ca29a894149451

[DecipherIt]

figured it out, it was an error on my end.

Still in trial but it PPed some files, so who knows it might just work now, or not lol

[DecipherIt]

Something still seems to trigger PPing without me asking for it. Scan is off in SG but

19:09:48 INFO HOMEPROCESSMEDIA^WEB_10 :: Processing... [Judas] Overlord - S03 NCOP.mkv
19:09:48 INFO HOMEPROCESSMEDIA^WEB_10 :: Analyzing name None
19:09:48 INFO HOMEPROCESSMEDIA^WEB_10 :: Analyzing name '[Judas] Overlord - S03 NCOP.mkv'
19:09:49 INFO HOMEPROCESSMEDIA^WEB_10 :: Analyzing name 'S3 Extras'

Happened without notice.