-
-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve startup #13348
Improve startup #13348
Conversation
This PR is an attempt to improve performance when loading torrents into session (especially when restoring a session, so many torrents are loaded in a row). @jagannatharjun, @An0n666, ping. |
I can test on windows if someone can provide me with a test build. |
37aa89d
to
db3bd7a
Compare
https://drive.google.com/file/d/1sZNbT5tvoPP2d_wVTFQ3mohJ1-Y-ZPr5/view?usp=sharing |
Thanks, beat me to it. Is this before or after the force push from 37aa89d to db3bd7a? |
After the force push(db3bd7a) |
Ok so I ran multiple types of tests with 5000 torrents on a SSD storage. First I tried with the last build that ned-martin used. #12825 (comment) It took 13 seconds for the UI to become responsive(white blank UI to transfer list) with only 1 non-working tracker. It took 2 mins and 29 seconds with 10 non-working trackers. Then I tried with the build provided here #13348 (comment) It took around ~10 seconds for UI to load and display no torrents. But then it went back to Not responding for another 2 mins and 40 seconds. So ultimately the UI remains unusable. I'm not seeing any improvement here. Except now the UI loads earlier with no torrents at all and it remains non responsive for almost the same amount of time. |
Well, I will continue further research. |
The load time is faster with less trackers. |
db3bd7a
to
2e6236e
Compare
Windows test build of 4.3.0(Alpha1) with listed libraries:
|
With zero trackers, session restores completely within 8 seconds. That is 5k torrents get loaded within 8 seconds. With 10 non working trackers in each torrent, it takes around 2 min 40 seconds for the UI to go from Not responding to a choppy one that lags like hell. The memory usages gradually climbs up to 800 MiB + and 50% of all 4 CPU cores were being utilised constantly during the Not responding period. The memory seem to get freed pretty slowly. |
Ok so I think I got something: The trackers that I was using were like this: I changed them to So I think something is breaking due to not having /announce in the tracker URL? Edit: The issue could be at qBt end as well. Like how the TRACKERS section parses the announce URLS. @glassez |
Did you double-check that the locking occurs without this patch even when using |
Checked 3 times. The patch actually doesn't improve anything. |
There's the white loading UI for 10-15 seconds in this build with /announce appended. On this build #13348 (comment) |
2e6236e
to
90b66ec
Compare
@An0n666 are you testing with "qBittorrent defaults"? For testing purposes: |
has anyone profiled the process during the time of lag/stall/choppiness? Perhaps a full alert log might help too, but it would still be trying to deduce the cause from bits and pieces, rather than actually looking at the cause of the lag in a profiler. |
e01a614
to
ecf8c37
Compare
@An0n666 to get the full alert log, you probably want to attempt to reproduce this with
qBittorrent will end up trying to scrape those URLs due to the auto scrape interval configuration, configured here: qBittorrent/src/base/bittorrent/session.cpp Line 1033 in defdd51
The results are processed here:
Would that be the problem? These URLs ( |
scrape attempts for HTTP trackers whose URL does not end with |
One difference that I forgot to mention between /announce and w/o /announce case. |
@An0n666, are all 5000+ mentioned torrents that you use in tests really running (i.e. started in non-paused state)?
IIRC, all your torrents have the same 10 trackers. Am I right? |
Yes. Running in downloading state(stalled). |
Yes same 10 http trackers without the /announce part in their URL. |
@An0n666
|
Ran the same test with torrents with /announce appended to tracker.
I have found more interesting thing: But without /announce appended, it announces to all trackers at once...skipping the looping. I think this generates 5000*10 = 50000 alerts at once per endpoint. And this is causing the locking issue. |
AFAIK, it is redefined in qBittorrent. But why do you ask about it? |
qBittorrent/src/base/bittorrent/session.cpp Line 1323 in 2bfaa82
|
Just for myself to better understand the internals of the code & also if there was anything there that could've been tweaked. |
memory priority was changed previously...could the same be done for I/O priority?? I'm using a program called TaskExplorer v1.2.8 to change (increase) the |
Well, I don't see much excitement... Personally, I don't like any of these options. |
What about the cases where people have 100+ trackers per torrent in different tiers and all announce at once at startup? In such cases the GUI will take couple of minutes to load. |
If I understand correctly, your comment refers to my last suggestion to restore the session behind the scenes, displaying only the progress indicator, am I right? |
I mean process the initial tracker alerts behind the scene as well, not just restoring the session. |
here's a crazy thought: What if the session is paused on startup, all torrents are added, then the session is resumed. Perhaps that will mitigate an "alert storm" to some extent. Clearly the torrent_added alerts would still be posted. |
When the torrents are being resumed after restoring the session, they could be resumed in batches, to further reduce the rate of alerts. Otherwise the user may experience another lock after the session is |
It's not such a crazy idea. At least we're crazy the same way, since it also crossed my mind. But I haven't had time to study it in detail yet. |
Don't forget that different alerts have different processing time. |
In addition, I noticed that tracker event processing is too wasteful. It affects the GUI with every related alert handling, although we could only do this with regular updates. |
7701a0f
to
dd1e7d0
Compare
@glassez just as a follow up really....do you want me to provide another test build? I didn't get a ping from you, so was just wondering..... |
I haven't added anything new here since. |
I have identified 4x users through the issue tracker that have Would be interested to see feedback from the user with @glassez @FranciscoPombal @thalieht What do you guys think? (these users have experienced the white screen/delayed startup etc) I can provide a build with |
Please take a look at another approach! |
@glassez how about transferring alert handling into a separate thread, my main objective is snappy GUI with no performance lost on libtorrent side and I suspect most of the slow GUI issues can be related to alert processing (I need to test it but I am too lazy). |
calling |
👍
Core processing of alerts is mostly fairly trivial and cannot cause too much load, so we are unlikely to benefit from transferring it to a separate thread, but on the contrary, we can only lose due to multithread interaction. |
No description provided.