qbittorrent-nox 4.1.6 corrupted fastresume data causes webui to hang

[pyr0ball]

Please provide the following information

qBittorrent version and Operating System

qbittorrent-nox 4.1.6
Ubuntu Server 16.04 (also tried upgrading to 18.04, same issue)
Server is running inside a Virtualbox VM container on host system, torrent content data is saved to host system via NFS/CIFS mounts

If on linux, libtorrent and Qt version

libtorrent: 1.2.x
cant remember Qt version offhand, not sure how to check from cli

What is the problem

I tried posting this to the discord and forums first, but got no responses, and after checking all of my filesystems for errors, I'm fairly sure this is a bug.

My instance was working fine up til I rebooted my server last week. Behavior is that the page just sits waiting to load. Chrome and firefox both showing the same behavior. Developer console network just says "pending". if I kill the qbittorrent-nox service daemon, it gives me an expected error, but when it's on it's just loading.

I thought maybe it was trying to run an extract on a big torrent or something so I disabled the external post-download script, but that had no effect.

Eventually I started poking around in the fastresume files thinking one of the torrents had gotten corrupted (I've seen that behavior with rtorrent in the past) and it turned out that was the cause.

When qbit is in a bad state, the service appears to be running fine, and I'm seeing no errors in the logs other than a couple of "files missing torrent has been paused" outputs.

In order to identify the torrents causing the hang I ended up having to move all the fastresume files out of ~/.local/share/data/qBittorrent/BT_backup and move them back in batches til I narrowed the issue down to the faulting torrent. Very tedious process when there's 800+ torrents in there.

In t he last few days, the number of faulting torrents has increased a ton, probably due to the number of torrents accumulating in my client. New corrupted torrents seem to happen if I leave my client going for a day or two and then restart the VM server. It doesn't happen every time, but it's becoming more and more frequent.

Anyhoo, I'm wondering if there's a way I could either get more debug output from qbit so I can more quickly identify faulty resumedata, or maybe some way of scripting the process so I don't have to spend an hour doing this when it happens? I think ideally what should happen is the web GUI should still come up, but show an error on the faulting torrent so it can be manually handled as needed

Steps to reproduce

Add 700+ torrents to qbittorrent-nox, hard reset of system or reboot without stopping service
With more torrents, simply letting the torrents seed for a few days then restarting qbittorrent-nox service seems to cause faults.

[Kolcha]

+1 , I also observed similar situation and even worst with fastresume files. but in my case qbittorrent process just crashes on startup until I remove damaged fastresume file. I also have ~1000 torrents and run qbittorrent from master branch and libtorrent 1.2 and that happened after system reboot. as I saw in logs qbittorrent didn't stop in 90 seconds and system just killed its process, so it is expected to get some damaged fastresume files. moreover, I see no difference between damaged and normal files... my python script (uses libtorrent) decodes it with no issues.
as solution I just reverted to 4.1.x branch and libtorrent 1.1, the issues are gone. I think qbittorrent still has not so good libtorrent 1.2 support (last used ~2 months ago) or libtorrent 1.2 is slightly buggy.
P.S> fastresume format changed during libtorrent upgrade and it is not backwards compatible, so after reverting it everything must be rehashed...

[pyr0ball]

@Kolcha did you have a way to pinpoint the faulting fastresume files? In my case, the process itself doesn't throw any errors or faults, it just hangs on webUI, and nothing in the logs show as anything is wrong.

[Kolcha]

unfortunately no. I also found no easy way to detect broken fastresume files... I found nothing in qbittorrent logs or any difference in files that cause crash and other "normal" files... so that's I reverted to earlier version.
as workaround you can just copy BT_backup folder before shutting down your server (even with working qbittorrent), and when such damage happens just kill qbittorrent process and restore BT_backup folder from your copy. this should work, I did (and sometimes do) this trick many times.
Especially when restarting qbittorrent/system in about few minutes after start. in such case almost always something damaged appeared. but 4.1.6 with libtorrent 1.1 not crashes, just may start rehashing a lot of torrents, I think this is normal, but will take a lot of time in my case. I know nothing what master branch version will do in this case. That I saw was a long time ago, so many things may changed.

[pyr0ball]

@Kolcha do know if there's anything I would need to back up to keep my current torrents and settings if I want to downgrade? I'm currently using the stable PPA install

[Kolcha]

nothing must be backed up. just leave qbittorrent config/data folders as is. all app settings will be restored (this is not a problem if qbittorrent config has unknow keys, they just ignored), but all torrents rehashing can't be avoided due to different fastresume format. all stats (i.e. ratio and similar things) will be saved.
only one drawback, just rehashing will take a lot of time... in my case that was ~10TB of data and rehashing it took about a day (~20 hours).

one more suggestion: change (decrease) fastresume save interval in app settings (not available through WebUI at least on 4.1.6, manual config edit required). default value is ~1h (don't remember exactly), I suggest to set it to 15-20 minutes. This will also mitigate the issue.
config file location: ~/.config/qBittorrent/qBittorrent.conf
key need to add or modify: Downloads\SaveResumeDataInterval=15 under [Preferences] section
value in minutes, qbittorrent must be not running.

[Chocobo1]

It would be very beneficial if someone could post a stackdump (by gdb) when the crash/hang occurs.

I think qbittorrent still has not so good libtorrent 1.2 support (last used ~2 months ago) or libtorrent 1.2 is slightly buggy.

IMO libtorrent 1.2 support is pretty much functional on qbt master branch, it shouldn't be any worse than using 1.1... unless there are new bugs in 1.2.

[pyr0ball]

@Chocobo1 If you could point me at a wiki or tutorial about how to get a stackdump, I'd be happy to provide one. I saved a few of the corrupted fastresume files in order to be able to reproduce the issue on demand

@Kolcha I'll try mitigating for now, see if that helps, thanks for the suggestion!

[Kolcha]

@pyr0ball you can follow this guide to get call stack. nothing special required. it is much better to do that on debug targets, make sure that -g option during qbittorrent compilation, and NO -s option on linking stage. -s option is set very rarely, same effect usally achieved during make install-strip, so just try "as is", even without -g option you still can have useful information, but with -g (debug symbols) you will be able see source line numbers in stack trace or even view source where crash happened :). in worst case you will see a lot of lines with question marks ??() instead of function names. then you will need rebuild.
libtorrent compiled without any special options (i.e. ./configure && make && make install) has debug symbols, but I cann't say anything about qbittorrent. I use script which builds qbittorrent and its dependencies (Qt and libtorrent) in way what I want, so I just don't remember default options.
I also will try to reproduce this, but cann't promise anything, I have no broken files and don't want to shutdown my "production" system for some time, many users depend on my server.

[Kolcha]

@pyr0ball one more note. if you have ~1000 torrents I almost sure you are reached open file descriptors limit (if you didn't change it already). this is 1024 by default (per process). qbittorrent opens a lot of files (and sockets) during startup, and 1024 limit was reached even when I had only ~450 torrents. It is very easy to change this value, it can be changed globally (i.e. any process will have new value) or for specific process, I prefer the second option. set it to at least 16384 (16k), this must be more than enough, I set this value to 65536 (64k) and now I run almost 1600 torrent without any connections limits.

[pyr0ball]

@Kolcha I already did increase the open files limit to 64k, so that's sorted.

Do I need to uninstall the PPA version, or can I compile in-place and just run it independently?

[Kolcha]

uninstall is not required, manually compiled version can co-exist with installed one, but after make install compiled version will take priority since it located in /usr/local/bin. so installed version become useless. if you don't want to build libtorrent - just install dev package from ppa.

[pyr0ball]

I've built libtorrent from source before, I think I can handle it. I'll get a debug build set up and see what I can get out of it

[pyr0ball]

Sorry for the lack of updates. I found that the fastresume data from previous torrents can actually be added back in if I do them in small batches and restart the service twice (must be at least more than once)

[ghost]

This issue is most likely another duplicate of many such issues posted in the past.
This is most likely the WebUI crash happening due to missing tags or categories from config.
This has been fixed in latest releases.

[Gbps]

Incredibly annoying issue, still happens on latest release if there is any error at all parsing any of the fastresume files. It also seems to nondeterministically fail as the threads doing the checking are exiting. The only way to fix this is to go through the log each time the process starts and remove every failing torrent's fastresume file from the filesystem.