[Search engines] "Failed to load the torrent [...]" on download from third-party SE (only since v3.3.2)

[nbusseneau]

Hi! I have a peculiar issue to report.

Problem

When trying to download a torrent from a third-party search engine, the following appears in an Invalid torrent message box: Failed to load the torrent: expected value (list, dict, int or string) in bencoded string.

This only happens with my third-party search engine for rutracker.org (I tested The Pirate Bay and Kickass Torrents plugins and there's no problem), and only with v3.3.2 and v3.3.3 (it is working perfectly fine in v3.3.1).

Reproduction steps

1. Install v3.3.1 (Windows link here).
2. Install rutracker.org plugin (use PluginTest for both username and password).
3. Launch qBittorrent and try to search for debian wheezy. 2 results from rutracker.org will appear.
4. Try to download any of the 2.
5. The download confirmation dialog opens without issue.
6. Now close qBittorrent and install v3.3.2.
7. Repeat steps 2 to 4.
8. Instead of the download confirmation dialog, the error message appears.
9. The same behavior happens in v3.3.3.

"This sounds like a problem with your plugin, not qBittorrent"

That is indeed possible. However I've spent the whole afternoon wrapping my head around this and from what I've gathered so far my plugin works as intended. Here are my arguments:

• If we launch python rutracker.py, a small test scenario will download a torrent file. Upon inspection of that torrent file, it appears to be perfectly well-formed. In fact, feeding it to qBittorrent by using "Open with..." will open the download confirmation dialog, as expected.
• If we launch python nova2.py rutracker all debian wheezy, take any download URL from the results, and input it to python nova2dl.py http://rutracker.org <download_url>, the torrent file will also be downloaded without issue and can be opened using qBittorrent.

This leads to me believe the issue lies somewhere beyond the scope of my plugin: it seems to return perfectly valid torrent files, but for some reason, since v3.3.2 these files are not considered valid anymore when downloading through the GUI (though the official plugins are still working and manual torrent addition work, as we saw).

Any ideas? I've started to look at the differences between v3.3.1 and v3.3.2, but I must admit I'm far from an expert of the qBittorrent codebase (never even compiled it).

[nbusseneau]

Apologies for the multiple title edits, I suck at titles... :(

[DoumanAsh]

Now that sounds curious.
I myself didn't look into C code for a while, but lemme see if i can find out how downloading is happening.

P.s. RuTracker for some time offers magnet links. I found it might be more useful to use it rather than torrent files that requiring downloading.

[nbusseneau]

Good idea, however magnets are not displayed in the torrents list when you make a search on the tracker. This would force the plugin to first access the description page of the torrent and then retrieve the magnet.

It's possible but I'd rather stick with downloading actual torrent files rather than HTML pages to be parsed for magnet links : a typical torrent file is something like 0.60KB whereas any rutracker.org HTML page is 20KB+. Sounds like it defeats the purpose of choosing magnets over torrent files to me.

In any case, thanks for having a look :)

[DoumanAsh]

Now it seems there were quite a lot of changes and this particular problem is related to our DonwloadManager.
It seems we no longer use python to download torrent files.
Instead there is own Manager that processes link to download.

And the error you see seems to be caused by somehow corrupted torrent file?

@glassez @sledgehammer999 There is just alot changes were happening with downloading. Care you help with that one? It seems likely that our download manager might now work perfectly fine from time to time

@Skymirrh how well your engine works with python3? I'm not getting any results at all
Or maybe there is encoding issue as my system uses japanese locale...

[nbusseneau]

The engine works with both Python2 and Python3, at least on my end:

Same results when moving Python2 up the path, except that qBittorrent will use Python 2.7.10 instead of Python 3.5.0.

Did you edit username and password at the top of the file? Set them both to PluginTest for testing. You can also just run python rutracker.py and see if any Python error occurs (you will need to move rutracker.py from %localappdata%\qBittorrent\nova3\engines to %localappdata%\qBittorrent\nova3 before running this).

EDIT: As for encoding I'm not sure, best would be to run python rutracker.py to check. I did add workarounds to work with rutracker.org encoding (CP1251), so there's that.

[DoumanAsh]

I cannot run it from console because python fails to encode correctly from jp :(
I'll try later to deal with it

[nbusseneau]

As for your question: And the error you see seems to be caused by somehow corrupted torrent file?
I don't know. The error message is apparently related to libtorrent, as it can be found here under enum error_code_enum. However the file that is downloaded using Python is perfectly valid, since if you manually input the temporary torrent file created by the engine to qBittorrent, the torrent will be added without problem.

If the DownloadManager does no longer use Python to retrieve the files then the error is entirely normal: rutracker.org requires a logged on user (so, valid cookies) and additional POST parameters for the download URL to be correctly retrieved. The DownloadManager has no way to know that since it is the duty of the search engine plugin to provide them, as is stated on the search engine writing tutorial:

def download_torrent(self, info):
     # Providing this function is optional. It can however be interesting to provide
     # your own torrent download implementation in case the search engine in question
     # does not allow traditional downloads (for example, cookie-based download).
     print download_file(info)

I guess the same problem will be encountered for any private tracker using user authentication.

[DoumanAsh]

Sorry, it was more like self-question.
The corruption is detected during parsing file.

But yes, i forgot about RuTracker rules to prevent non-users to access torrents...
That's then the reason.

But i'm unsure how to approach it.

It is i suppose incorrect that we do not rely on python download anymore.
@sledgehammer999 your opinion?

[nbusseneau]

Yea, I think private trackers with authentication were kinda forgotten when the new DownloadManager was introduced, haha :D

Not sure how to proceed either, but to me it sounds like the search engine should still be responsible for providing the torrent file. Due to the nature of private trackers it sounds difficult to implement a generic DownloadManager with hooks for cookies, POST parameters or whatever the private tracker fancies using.

[glassez]

DownloadManager now supports cookies. But some improvements are still not fused (see #4670). So soon we will not need to use these "dances with a tambourine" (I mean separate python download routine) to download torrents with private trackers.

[glassez]

@Skymirrh, your plugin is really needed to pass something through the POST params, or GET will work too?

[nbusseneau]

It doesn't seem to work when using a GET request, rutracker.org requires the POST param.

EDIT: Wait, I had messed up the cookies. It does seem to work with a GET request. So maybe that would work, however it will require a test with DownloadManager using cookies to be sure.

[glassez]

@Skymirrh consider #5026 and #5085

[nbusseneau]

@glassez Thanks for the heads up. This temporary revert will indeed fix the issue for now. I am looking forward to that new plugin system and will gladly adapt my plugin accordingly whenever it becomes available :)

[glassez]

I am looking forward to that new plugin system and will gladly adapt my plugin accordingly whenever it becomes available

You not worried that it will not use Python for scripting?

[nbusseneau]

Well I do prefer Python but if the scripting system is convenient enough and can accommodate cookies and POST param easily... why not?

[nbusseneau]

Closed in a9e6328.

[SatenderRawat]

I have done this method just now on my Ubuntu 16.04LTS & got 100% result.

1. First hit search button then hit search plugins/engines.
2. Simply hit check for update & within few seconds you will get a message " All your plug-ins are up-to date".
3. Search for any subject & I am 100% sure your problem must be resolved like mine.

[nbusseneau]

@SatenderRawat I fail to see how that is relevant to the discussion at hand (which was already closed). Are you sure you haven't posted in the wrong issue?