This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
GUI performance with many thousands of torrents - transfer list, icons, and more #13304
Comments
FranciscoPombal
added
Confirmed bug
May have some relevance?! |
|
The only thing I can think that is regularly called and can cause this is transfer list refreshing, you can change the interval in the advance setting. Actually, in 4.2.5, subsequent refresh calls can overlap (fixed in master) when CPU is busy causing more damage, so setting large refresh intervals actually may help. AFAIK you can also disable speed graphs from advance settings if you think that is the problem. Another possible reason can be that libtorrent is posting a large number of alerts for some reason, you can add some logging and take a look. Also, I don't think Qt can be the reason. Qt is very efficient and is used in many other high-end applications So it must be something within qbittorrent itself. |
Is the transfer list all refreshed at the same time? Is it possible to perhaps optimize it by just refreshing the visual representation of the items that are visible at any given time? Or do we already do that? As mentioned in #11822 (comment), could it also be the case that there is room for optimization in the way the status icons are displayed in the list? My suspicion is that a bunch of svg icons frequently being loaded could be a problem, perhaps.
A bit of anecdotal evidence: I have noticed, particularly in low end machines, that qBittorrent's WebUI becomes unusable (unresponsive) under high resource consumption scenarios. I suspect this is due to the fact that all parts of the application are fighting unconstrained for a share of the computing resources. Ideally, computations related to GUI code, WebAPI requests (such as those coming from the WebUI), etc should have much higher priority/precedence than, say, libtorrent piece hashing, so that the user can still control the application, even at the expense of a little transfer performance while doing so. I have no idea how this could be implemented, though.
Indeed. |
|
Perhaps qBittorrent defaults need to be adjusted again?!
In relation to progress bar would the use of palette colors like the Another thing, that may have an impact is if torrents are checking on start-up....currently if a torrent is checked/re-checked, there is no record of this being carried out in the execution logs. |
They do update at the same time but the real implementation of torrent update in libtorrent is actually asynchronous and shouldn't block GUI, even with 10000 torrent I don't see it causing too much load to block GUI indirectly. I think rather then speculating and wasting time it would be better if someone can replicate and profile the application.
Icons are cached. |
Ok, not a problem then.
Well said. An idea could be comparing two profiling runs, one with transfer list icons, and one with no transfer list icons at all, just to see how much CPU the icon display code eats up. I can probably do that in the next few days, if no one does it before that. |
transfer list icons are cached, they can't be the problem, they are only loaded once, but you can try benchmark over it if you wish. |
|
This may also help #13316 |
|
off topic, where does qBittorrent make use of |
|
https://github.com/qbittorrent/qBittorrent/search?l=C%2B%2B&q=zlib Keep in mind the results preview in GitHub's search GUI only shows the first couple matches in each file. |
|
I'm very sorry but my problem is very similar and was never fully resolved. I'm seeking help and if anyone wants, I can perform tests or benchmarks etc.
The most serious problem is RAM consumption. Sometimes it can stay adequate (2-3 GB) for weeks, and sometimes it's starting to eat more and more by a minute, and when it reaches >20 GB other apps start crashing. I noticed once again that no matter what, on startup qBT seems to access downloaded files of paused and finished torrents, even though there should be no such need. Another suggestion about GUI performance is about tags (or categories). I noticed that whenever current list holds smaller amount of torrents, it performs much better. By default the GUI loads always in "All" category, so it always forcing the worst scenario of RAM consumption from the very start. As soon as it becomes responsive I switch to another category. My guess is that if it's possible to make qBT start with an empty category, or a category with just a few torrents, the GUI would load much faster. I'm not sure what I should provide if you ask me for "Full qBittorrent logs and settings in plain text". My log files are always huge (several MB) and quite useless. Also unicode torrent names are not properly saved. The settings file include some bits that I'd like to keep private, e.g. cookies and paths. |
|
Right now it appears to be working well and responsive but RAM usage is constantly increasing. From 3 GB ~3 hours ago to 7.5 GB now. I didn't do any actions with existing torrents and didn't add any new ones. 23 MB downloaded and 5 GB uploaded. |
|
That's a very old version and a whole lot of torrents indeed. I would suggest backing up all your settings/fastresumes (just in case, see https://github.com/qbittorrent/qBittorrent/wiki/Frequently-Asked-Questions#Where_does_qBittorrent_save_its_settings for more info), and trying a build of recent Many important fixes have been deployed since 4.1.9, including fixes that massively improve memory usage and GUI startup time in some circumstances. Recently a user with many torrents reported a massive improvement with recent commits: #12825 |
|
Some general news: #13348 has been submitted with some improvements to startup code. Furthermore, an issue has been identified with tracker announces that can impact GUI startup time massively. Investigation is still ongoing. |
|
I will try that but which file should I use? What's the difference? Also will backing up settings/fastresumes really help to avoid massive rechecking in case I downgrade back? |
No real difference for our purposes here. Just use
Yes. |
I've tried this. My observations:
I'm already used to confusion about reading tons of files it wouldn't do anything about (paused torrents will not be downloaded and will not be uploaded). But again I'd like to know if there are any plans to fix this behavior. |
This may be the result of a "first start" with a version >= 4.2.0 when coming from a previous version. The fastresume file format changed between versions, so it is expected that some rechecks/fastresume re-saving takes place. From the news section of the website (version 4.2.0):
This is also why I advised you to backup your fastresumes before trying out the new version - in case something catastrophic happened and you had to roll back, it would save you a lot of time rechecking to roll back fastresume formats. So, it would be interesting to know if this behaviour is reproducible across more than one startup/full close cycle. Assuming you always let qBittorrent exit cleanly between runs (i.e., don't force terminate it), I wouldn't expect those same >2000 torrents to keep rechecking every time.
It is possible that this is the result of changing over to the new .fastresume format. It is also possible that there is some other problem with these torrents. If you experience frequent rechecks/redownloads on startup on Windows, even after making sure that qBittorrent has had the opportunity to save the fastresume files in the correct format and exit cleanly, it is usually due to one of these reasons:
It is possible all the rechecks and fastresume conversions in this "fresh start" played a role in this, I would expect the startup time to have decreased, all other factors being equal. At least it's nice to now that it's not worse than before. Recently, there have also been some recent developments related to improving startup time in #13348
👍
If you mean the .fastresumes, it is inevitable for qBittorrent to read those, even for torrents that are stopped. Otherwise, it can't know the information it must show in the transfer list. That being said, it is possible that there is much room for improvement in the code related to reading fastresume files on startup. |
Okay trying it second time as I'm typing this. And yes, I do not terminate qBT process. Just choose Exit and wait for it to close properly. Notes about the first run:
The GUI loaded now and I didn't notice much of improvement in loading time - only about 10 minutes less. And those >2000 torrents are getting rechecked again. So it will happen again on each application startup.
No, .fastresumes are about 500 MB in total, it wouldn't take long to read them. I mean reading hundreds of GBs of downloaded files across my download folders. |
qBittorrent in general most certainly supports Unicode. What is your OS? Are you running a recent Windows 10 version? Are your locale settings correctly configured? It is also possible the fastresume files for those particular torrents are corrupted. It would be nice if you could open a separate issue report for this. If possible, upload both torrents in a .zip, so that others can test on their systems to rule out the possibility of some problem/misconfiguration on your end. But first let's wait for the results of the second run.
Do you have
Try resuming and pausing, and force rechecking, and see if the problem persists in the next startup. If it doesn't work, you can always remove it and re-add it.
Strange, I believe all should be touched on exit, or every X (60, I believe) minutes, even if they remained paused during the session. Other than that there are a couple of actions that will force the .fastresume to be overwritten. Resuming/pausing is one such action, I believe. Just to be sure, you can leave the client running for a couple of hours and see if all .fastresumes are updated then. |
8.1 x64, and most certainly there are no problems with my locale. But I see that there are only question marks in qbittorrent logs for every unicode character in torrent names. They are displayed and operated just fine in qBT list, yes. But something seems off when qBT tries to (a) display a non-Latin OS-reported error in the list and (b) write unicode data to logs.
I do but I don't think there were such folders for those torrents. Should I try removing all of .unwanted and .parts files?
I set it to 480 minutes because at some point I lost an SSD to qBT. #9152
There is no way rechecking speed of tons of GBs can be improved. I think removing the need to recheck that much data is the only good option... |
Echoing my request from the previous post, please submit a new issue report with example torrents where you see this happening.
The
The conclusion from that discussion was that .fastresume writes were most likely not the cause of death of your SSD, as I recall. Regardless, that was back when the save interval defaulted to 3 min. The new default has been 60 min for some time, which is 20 times lower - more than adequate. 480 minutes is overkill. |
|
I'm still sorting out the the changes that came with the first run of new qBT version. I'm seeing that in many cases empty folders I created myself were deleted. E.g. there was a fully downloaded / finished / paused torrent with 1 folder and 3 files inside of it. After it was completed I created a subfolder inside it with specific name, so the torrent folder had 1 subfolder and 3 files. Now the subfolder is gone. It seems to me it was really removed by qBT, since I'm keeping backups of folder and file structures and there was no such differences before. So it looks like qBT started to treat folders of downloaded files as exclusive territory. Is this a thing? Could it be related to handling of outdated .unwanted subfolders?
I re-added the affected torrents and will see if it happens again.
I've tested with SSD monitoring tool and changing that interval massively affected the amount of data written to disk each 24h. I don't agree with any long interval being an overkill because it really doesn't affect anything (but short interval is an overkill for write-sensitive hardware). On my PC, qBT is running for weeks and months with probably only couple of dozens restarts per a year. Why would I want to write fastresume data frequently if it's being kept in RAM already? With 8h interval, in case qBT crashes, I'm most likely going to rebuild any lost metadata about recently added or downloaded torrents without problem. |
|
After running for more than 24h, I restarted qBT again.
|
Including no more weird text/Unicode errors? Nice, that's good to hear.
Hmm, I don't think this is suppose to happen. IIRC we still update .fastresumes even for stopped torrents every fastresume save interval. @glassez can you please confirm this? Have you let it running long enough for at least a .fastresume save interval to occur? In your case, it would have to be over 8 hours (I would do 24 or a bit more just to be safe). Can you please open a new issue and post a couple of fastresumes + their corresponding torrents for the torrent that are stuck in that recheck loop plus and a couple of fastresumes + their corresponding torrents for torrents that are not affected by the issue? My suspicion is that they will be quite different and the cause of this issue will be exposed by that difference. Alternatively, you can remove them and add them all again... but that is probably quite tedious.
I'm interested in knowing what the startup time will be once all other outstanding issues are resolved (right now, that would be the recheck loop issue + the fastresumes not updated issue, which is likely related to the former). |
So far this endeavour has lead to the fixing of a bug in libtorrent that was causing massive increases in startup time if torrents had trackers with URLs not ending in So please test with the build linked in #13348 (comment). It would be interesting to see whether or not you see an improvement in startup time with these new build, even without having solved the recheck problem as mentioned above. |
I tried this first. Observations vs.
So I identified 2 different torrents.
Should I upload .torrent and .fastresume files related to both torrents into new issue? |
Yes, please. As for the rest of your findings, I would suggest the following: backup the fastresumes of the problematic torrents, and remove them from their original location. The goal is to try to get a clean startup shutdown where no torrents enter an error state/recheck loop. Then we can fairly judge possible advancements made in startup performance. The errored fastresumes are a separate issue most likely, that should be handled separately. |
|
Done.
Sorry but trying to do anything with that amount of torrents (files) is very problematic. Maybe it would be possible if I could somehow export a list of folder paths from selected torrents. |
No problem I completely understand. Your collaboration and diligence is much appreciated.
It's a bit unclear to me what you have in mind, but if you want to extract paths, you can probably do that with the WebAPI and some scripting: https://github.com/qbittorrent/qBittorrent/wiki/WebUI-API-(qBittorrent-4.1) Look into the torrent list and torrent properties API methods. |
|
Honestly, I get the GUI to freeze with just a few hundred torrents. This is the #1 issue that needs to be addressed. |
|
Here is my report after upgrading to 4.3.1 - #13416 (comment)
So, not much change GUI-wise. It's still frozen while starting up. RAM usage seems abnormally small for now, at ~430 MB. |
|
Would someone performance test this version of web ui with a couple of thousand items too? Read more about the progress here. |
|
I'm getting a really bad lag issue on macOS on 4.3.2. The UI lags a lot when scrolling. I checked my CPU usage and it jumps up to 100% whenever I scroll. This leads me to believe that the app is doing something extremely inefficient (maybe refreshing the scroll view many times when scrolling) resulting in this high CPU usage and intense lag. This is a little disappointing given that qBit is supposed to be a relatively "lightweight" client but is actually eating up CPU and battery when doing basic scrolling in the app. But I don't blame the developers because I'm sure they would have fixed it if they knew the issue. |
|
I don't know how similar this issue is to #12900, which is specific to Macs, but I am mentioning it here just in case it is related in any way. I will also copy and paste a comment I made on that issue. The UI will start to lag more the bigger the window size of the torrents. If you decrease the number of torrents that can be seen at one time, for example by clicking on the content pane or by making the window smaller, the lag becomes less harsh. This is likely due to inefficient rendering of the table views which make up the torrent files in the GUI list. I see a CPU spike of 100% when scrolling in qBittorrent [on Mac]. This should not be happening. When I scroll through a list in any other app, it barely breaks 20% CPU usage. I assume this is an issue in qBittorrent simply because it wasn't coded "natively" for the macOS platform. @FranciscoPombal If you would like me to perform any sort of tests, I would be happy to do so. |
|
@puddingsoda Thanks, but I don't have access to macOS. If you can debug/investigate yourself, if you want. Perhaps your findings would help shine light on other issues for other platforms as well. |
|
@FranciscoPombal I temporarily unpinned this in favor of the Poll issue. It was a random decision. If you want you can unpin #9849 and re-pin this. |
No problem, understandable. I wish GH allowed a few more pinned issues. Anyway, I've been using the Projects tab to track important/frequent issues. In fact, this one should have been there already. Will add now. |
|
Disk cache set to '256MiB' |
|
Well not sure if this is correct place to post but I will say that the qBittorrent web GUI fails on me if I get above like 1200-1300 torrents in the queue. I can add more using the API and it will respond to the add no problem. If I let it fall back down in to the 900-1000 range then the web interface will start working again showing torrents and all the information. If I am above 1200-1300 the web interface comes up but it won't show any torrents or upload/download total speeds. Every thing is just blank. I can still use the search tab. So the web interface isn't totally broken but it just doesn't like to handle a large number of torrents. This is for qBittorrent v4.3.5 Web UI (64-bit) on Linux. |
ghost
locked and limited conversation to collaborators
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
qBittorrent's GUI performance with many torrents (as in, many thousands of torrents) could probably be better.
I created this issue for the purposes of experimentation/investigation/discussion about this, without having to deal with excessive baggage from previous threads.
Relevant background:
TL;DR: a bug in libtorrent was causing a long-running operation to block everything else. While that has been fixed in libtorrent 1.2.8+ (report confirming: #12825 (comment)), the GUI performance still leaves something to be desired.
TL;DR: On Linux, investigation into the problem showed that inefficient WMs/compositors (or some inefficient way qBittorrent interacts with them!) play a significant role in the perceived lack of performance. Other than that, it was probably part #12825, part #11822 (comment):
This is not a thread for regular bug reports. It is expected that posters have some idea of where the problem might be or have done some preliminary investigation/analysis/benchmarks about the problem with developer/power-user performance profiling tools. This is the place to share such findings.
When posting in this thread and investigating these performance issues, please provide as much information as possible from the get-go, to make it easier for others to understand your setup ad environment..
At least:
callgrindandmassifare your friends.The goal is to identify specific points where performance can be improved in relation to the GUI, and particularly in the transfer list. As each specific cause/bug becomes clearer and defined as its own separate matter, discussion about it may move to a dedicated thread about it (this is also useful for having something specific to close when submitting a PR). This one is just for initial investigations/brainstorming, e.g. "What if excessive icon load/unload is slowing down the transfer list?", "How can we test that?" "We could try doing foo..." "I tried <this patch>, and got the following results..."...
Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.
The text was updated successfully, but these errors were encountered: