Change some settings defaults for better behaviour out of the box

[FranciscoPombal]

• AnnounceToAllTrackers (off->on): not sure why this was off by default. It should improve peer discoverability. The situation this change seeks to improve needs a better and more involved fix overall than this simple default setting toggle.
• CheckingMemUsageSize (16 MiB -> 6432 MiB): a 4816 MiB increase in memory consumption seems worthwhile for a nice performance boost in most cases.
• DiskCacheSize (64 MiB -> Auto): auto yields the best performance without committing to a huge fixed value.
• QueueingSystemEnabled (on -> off): this is a big one. Users often don't realize this is on and get confused as to why some torrents get the "Queued" status. By defaulting to off, several simultaneous downloads work out of the box. Users who know they want the queueing system will go look in the settings anyway. Already addressed in Disable Torrent Queue by default #11678
• SendBufferLowWatermark (10 KiB -> 1600 KiB): According to https://www.libtorrent.org/reference-Settings.html#send_buffer_low_watermark - "For good and snappy seeding performance, set this fairly high, to at least fit a few blocks." 1600 KiB is 100 blocks, which is "fairly high".
• SendBufferWatermark (500 KiB -> 16384 KiB): According to https://www.libtorrent.org/reference-Settings.html#send_buffer_watermark - "If set too small, upload rate capacity will suffer. If set too high, memory will be wasted." 16384 KiB (16 MiB) seems like a small amount of memory to give up in exchange for better upload capacity out of the box.
• SendBufferWatermarkFactor (50 -> 110): According to https://www.libtorrent.org/reference-Settings.html#send_buffer_watermark_factor - "For high capacity connections, setting this higher can improve upload performance and disk throughput. Setting it too high may waste RAM and create a bias towards read jobs over write jobs.". 110 is a tentative compromise to improve upload performance and disk throughput while keeping read bias in check. All changes related to default numerical values of send buffer settings required more discussion in its own PR/issue.
• UseRandomPort (false -> true): The initial port chosen by qBittorrent may clash with something else the user already has that is aways using that port (low probability, but still). Thus, qBittorrent will always fail listening on that port, causing unexpected problems for the user. Users who know they want a fixed port will go to the settings anyway.
  In short, with UseRandomPort = true; the port number will be assigned by the OS (randomly) which should be an unused port at the time making the API call.
• IPv6Enabled (false -> true): IPv6 adoption is increasing more and more. Turning this on by default should increase peer discoverability (because it enables connecting to IPv6-only peers). Dropped due to Rework the listening IP/interface selection code #11592

[FranciscoPombal]

These changes are suggestions from my experience with using qBittorrent "in the field". The numerical values, in particular, don't have a bulletproof justification, it's just a ballpark.

The idea is to optimize defaults for the common case, where it is understood that "common case" now also includes users with relatively fast connections (50+ megabits to 1 gigabit, my condolences to any Australians or Americans reading this) and machines (with SSDs, and the like). The goal is to do this without raising the minimum requirements to run qBittorrent with several torrents too much (I think the main concern would be memory usage).

Of course, as always, users at either extreme end (ultra fast machines and connections, machines dedicated to high speed seeding, racing, or users with very low available memory and other system resources) will need to tune their settings for their specific use case.

I would appreciate comments about all of these potential changes, but especially relating to the watermark* ones. Does the memory consumption of these settings increase proportionally to the number of active torrents or peers or is there a single send buffer for everything?

If it is the former, then at least the value for send_buffer_watermark is probably unreasonable and wasteful for everything except the fastest connections.

Additionally, how difficult would it be to detect the number of hardware threads on a system at runtime? I ask this because, like I said in a previous discussion, I think the default value aio_threads should be changed to be 4 * number_of_hardware_threads. The reason is that libtorrent will spawn 1 SHA-1 thread for every 4 aio_threads. Nearly all systems will perform better, particularly when rechecking, if libtorrent spawns as many SHA-1 threads as there are hardware threads available. Nowaydays most people have machines with at least 2 cores (laptops) or 4 cores (desktops), and many people have more than this. With aio_threads set to 4, libtorrent will only ever use one of them for SHA-1. But that is a matter for another PR...

@arvidn I would appreciate guidance, thanks

[Chocobo1]

Overall I'm a bit curious if updating those memory related tweaks will affect less powerful system / embedded systems such as raspberry pi? I mean I don't want it to crash because the default settings is too high for it.

[FranciscoPombal]

Overall I'm a bit curious if updating those memory related tweaks will affect less powerful system / embedded systems such as raspberry pi? I mean I don't want it to crash because the default settings is too high for it.

Yes, that is my main concern as well. I am open to suggestions regarding all of the fixed numerical values relating to memory usage, especially the send buffer stuff. Ultimately, I can remove those kinds of changes from this PR, but I still think the other changes (like AnnounceToAllTrackers) are worth it on their own.

[sledgehammer999]

QueueingSystemEnabled (on -> off)

Personally, I wouldn't like this change. However, if others agree I am not going to block this.

UseRandomPort (false -> true):

IIRC true was the default for some time. I can't remember the exact reason we switched back, but I think it had something to do with webui users (and console users).

IPv6Enabled (false -> true):

Not relevant anymore after my #11592 was merged.

For the others, I don't have any particular comment.

[FranciscoPombal]

QueueingSystemEnabled (on -> off)

Personally, I wouldn't like this change. However, if others agree I am not going to block this.

What are your reasons for not liking it? It would be nice to know. In any case, this change can be subject to a vote, it it is all the same to you.

UseRandomPort (false -> true):

IIRC true was the default for some time. I can't remember the exact reason we switched back, but I think it had something to do with webui users (and console users).

It would make sense to be true by default, because it is better for the average user while not making things more difficult for those who want it false. Users who want this set to false, i.e. webui and console users, already have to change the config to:

• set the port number to the desired one, as it is unlikely that qBIttorrent randomly assigns the exact fixed port number they want on first run.
• change a bunch of settings they would have to anyway (password, etc).
  These users have to edit the config either way, might as well make it explicit that setting a fixed port is intended for advanced usage.
  in light of this, if you also don't have strong feelings one way or the other, we could vote on whether this should be changed or not, like above.

IPv6Enabled (false -> true):

Not relevant anymore after my #11592 was merged.

Yeah I realized it became irrelevant as well I will rebase this on top of the current master and remove this change.

[Chocobo1]

It would make sense to be true by default, because it is better for the average user while not making things more difficult for those who want it false. Users who want this set to false, i.e. webui and console users, already have to change the config to:

Just to make sure you guys are talking about the same thing, the affected setting is for bitorrent port and webui always listen on a fixed port 8080.

[FranciscoPombal]

Just to make sure you guys are talking about the same thing, the affected setting is for bitorrent port and webui always listen on a fixed port 8080.

yes, and anyway webui users typically want to change both values. for "regular" GUI-only users, random listening port is better for the reasons outlined above, and the webui port is not important for them.

[FranciscoPombal]

Rebased and dropped IPv6 setting change due to #11592

[sledgehammer999]

About my comment for random port: I did some history digging and issue digging. I couldn't find the problem I was remembering. So there's a possibility I misremember this. Also after #10523 it probably isn't a big problem to go full random.

AnnounceToAllTrackers (off->on)

I missed this. Don't we break some kind of universal guideline on good behavior for clients if we enable this by default?

[LordNyriox]

@sledgehammer999:

Don't we break some kind of universal guideline on good behavior for clients if we enable this by default?

Does that matter, given that a significant chunk of your userbase seems to expect the behavior that enabling it would provide?

[FranciscoPombal]

Also after #10523 it probably isn't a big problem to go full random.

Does that matter, given that a significant chunk of your userbase seems to expect the behavior that enabling it would provide?

Yep, my thoughts exactly.

[ghost]

Queue should be disabled by default. Have seen a lot of users who switched from utorrent/bittorrent and are limited to only seeding 3 torrents at a time by default. Some don’t even understand that the torrents queued for seeding are actually not seeding.
What’s the devs logic to set such a low value? By default seeding should be unlimited (-1). However setting a limit on max active downloads is reasonable.

[Seeker2]

AnnounceToAllTracker should be left off, but announcing to 1 tracker on all tiers should be on.
Otherwise, this will serve to reduce or eliminate HTTPS public trackers which have far higher bandwidth requirements than regular HTTP trackers (let alone udp ones!)
The HTTPS trackers are far more likely to run into bandwidth hosting issues if every peer or seed starts using the HTTPS tracker as well as the regular tracker on the same tier.

Quite a few torrents are made with tracker tiers for a reason, and hammering all those trackers by default because of connectivity issues only some people have seems a really poor workaround.
If qBitTorrent is failing to switch to a working-and-reachable tracker on each tier when one is available in a timely manner (not taking 3+ hours just to switch between 3 trackers), perhaps that should be investigated and fixed instead.

I had to clarify working-and-reachable trackers, because many proxies do not work with common udp trackers due to them not supporting UDP packets.
...or ISPs that block/scramble unencrypted tracker packets, which might prevent HTTP trackers from working.
...or IPv6 trackers, if there are any.

[NotTsunami]

AnnounceToAllTracker should be left off, but announcing to 1 tracker on all tiers should be on.
Otherwise, this will serve to reduce or eliminate HTTPS public trackers which have far higher bandwidth requirements than regular HTTP trackers (let alone udp ones!)
The HTTPS trackers are far more likely to run into bandwidth hosting issues if every peer or seed starts using the HTTPS tracker as well as the regular tracker on the same tier.

Quite a few torrents are made with tracker tiers for a reason, and hammering all those trackers by default because of connectivity issues only some people have seems a really poor workaround.
If qBitTorrent is failing to switch to a working-and-reachable tracker on each tier when one is available in a timely manner (not taking 3+ hours just to switch between 3 trackers), perhaps that should be investigated and fixed instead.

I had to clarify working-and-reachable trackers, because many proxies do not work with common udp trackers due to them not supporting UDP packets.
...or ISPs that block/scramble unencrypted tracker packets, which might prevent HTTP trackers from working.
...or IPv6 trackers, if there are any.

Agreed. I am in favor of all changes except AnnounceToAllTrackers.

[FranciscoPombal]

Upon reading @Seeker2 's post and reading through #11279 one more time, I think I have changed my mind.

IMO, the correct fix would be to:

1. AnnounceToAllTracker should be left off, but announcing to 1 tracker on all tiers should be on.

(what @Seeker2 said).

2. Provide a more sophisticated tracker edit dialog, that clearly shows which tier a tracker is in and allows the user to move trackers between tiers and add trackers to specific tiers.
   Thus, if a user is having trouble finding peers, they can add a bunch of trackers each to a different tier.

Regarding point 2: The downside is that the current feature of "automatically add the following trackers to new torrents" would probably need an upgrade, to support specifying the tiering of the auto-added trackers.

It should be noted that, with the existing code base, no combination of options can bring qbittorrent to full compliance with BEP 12. What's more, BEP 12 is very specific about the announce order in each case, but doesn't specify whether a client should always announce to all trackers in a tier even if the first was successful.

[FranciscoPombal]

@Chocobo1 @sledgehammer999
Dropped the change to the AnnounceToAllTrackers setting in light of previous comments.

Should I remove the changes to send buffer stuff, which will probably require much more discussion? Since we all seem to be in agreement about the other changes, they could get merged right away. I was kind of hoping this would be able to make it in some form to 4.2.1, but alas.

[Chocobo1]

Since we all seem to be in agreement about the other changes, they could get merged right away.

I think you missed one change discussed here: #11637 (comment)

What about simply raising it to 32 MiB instead, as a compromise?

OK for me.

[Chocobo1]

Should I remove the changes to send buffer stuff, which will probably require much more discussion?

👍

[ghost]

Use random port should be off. As 4.2 addresses the default port 8999 already.

[ArcticGems]

* QueueingSystemEnabled (on -> off): this is a big one. Users often don't realize this is on and get confused as to why some torrents get the "Queued" status. By defaulting to off, several simultaneous downloads work out of the box. Users who know they want the queueing system will go look in the settings anyway.

Should this one still be included since it's already been done with this merged PR??: #11678

[Seeker2]

Another elephant-in-the-room concerning people having trouble getting torrents working is it's not very apparent why a torrent isn't downloading or uploading.
From what I understand, the only clue in the main torrent list window is the torrents with trackers errors show up as "stalled".

This post: #11320 (comment)
...doesn't seem to be a case of all udp trackers failing OR all apparent trackers on a torrent being tried-but-failing. qBitTorrent doesn't even appear to be trying on some.
The first torrent shows SEEDING and connected to 1 seed, which I've seen numerous times in my own use of qBitTorrent.
The middle torrent shows a poorly-created torrent with all the trackers in 1 tier. I manually split those trackers into different tier groups when I see them on uTorrent, but that's annoying to do in uTorrent and a HUGE PAIN in qBitTorrent. That's one of the few examples where it's really tempting to just announce on all trackers, but what a mess doing that! (There might be 100+ trackers on it!)
The follow-up post by the same person: #11320 (comment)
...shows qBitTorrent clearly failing on tracker updates where uTorrent does not.

[FranciscoPombal]

Addressed everything.

@Chocobo1

I think you missed one change discussed here: #11637 (comment)

Thanks, done.

@ArcticGems

Should this one still be included since it's already been done with this merged PR??: #11678

Done.

@An0n666

Use random port should be off. As 4.2 addresses the default port 8999 already.

???

@Seeker2

Another elephant-in-the-room concerning people having trouble getting torrents working is it's not very apparent why a torrent isn't downloading or uploading.
From what I understand, the only clue in the main torrent list window is the torrents with trackers errors show up as "stalled".

This post: #11320 (comment)
...doesn't seem to be a case of all udp trackers failing OR all apparent trackers on a torrent being tried-but-failing. qBitTorrent doesn't even appear to be trying on some.
The first torrent shows SEEDING and connected to 1 seed, which I've seen numerous times in my own use of qBitTorrent.
The middle torrent shows a poorly-created torrent with all the trackers in 1 tier. I manually split those trackers into different tier groups when I see them on uTorrent, but that's annoying to do in uTorrent and a HUGE PAIN in qBitTorrent. That's one of the few examples where it's really tempting to just announce on all trackers, but what a mess doing that! (There might be 100+ trackers on it!)
The follow-up post by the same person: #11320 (comment)
...shows qBitTorrent clearly failing on tracker updates where uTorrent does not.

Perhaps qBittorrent should add a new state, something like "Tracker Error", to better informed the user. And it also should not fail announces when other clients don't as well. Maybe some kind of timeout is set too low? Anyway, those problems are all out of scope of this PR.

[ghost]

Seeing some crazy disk I/O when diskcache is set to auto in a 32G system.

[arvidn]

@An0n666 "crazy" good or "crazy" bad? Is the disk cache itself being flushed to swapfile?

[ghost]

@An0n666 "crazy" good or "crazy" bad? Is the disk cache itself being flushed to swapfile?

crazy bad. It started reading at 150 MB+ from disk. No download/upload ongoing.
Maybe it was loading files onto cache from disk?

[FranciscoPombal]

Seeing some crazy disk I/O when diskcache is set to auto in a 32G system.

According to current libtorrent code, auto would set the cache to around 800 MiB on a 32 GiB system. This is probably excessive. You can manually set it to lower.

Meanwhile, libtorrent could set auto cache to be amount_of_ram/60 or even a flat value of, say, 500 MiB for systems with > 16 GiB of RAM.

[ghost]

Most people would not want to waste that much RAM on a torrent client. When people see that much disk I/O or RAM usage they’ll probably switch to some other client instead. Why not keep the default to a low value so everyone is happy? If someone wants to allocate more RAM they could change it from settings.

[arvidn]

I don't think 800 MiB is excessive. If one downloads a torrent, if the whole torrent can fit in the cache you get the best performance.

It's not obvious to me that setting the cache to a high value is inherently problematic (apart from the fact that the process will use a lot of anonymous memory, which won't get evicted gracefully under memory pressure).

@An0n666 could you provide more details of what is being read from disk?
is a file being checked?
is a file being moved?
you say 150 MB (I assume you mean per second), would you expect libtorrent to read faster or slower than that? (and why?)
what tool did you use to measure the 150MB/s, and what exactly did it measure?

[ghost]

I think the disk read maybe unrelated. It could be due to some file check on startup(even though nothing was saying checking).
800 MB is not too much but if your system is not dedicated to torrenting you might want other processes to use that extra RAM. Basically it all depends on priorities.

[arvidn]

800 MB is not too much but if your system is not dedicated to torrenting you might want other processes to use that extra RAM. Basically it all depends on priorities.

agreed. In fact, in the next major release of libtorrent (2.0) the disk cache will use the kernel facility, to make it use as much RAM as the kernel has available to it, and gracefully evict it when the system needs more free RAM.

[FranciscoPombal]

Most people would not want to waste that much RAM on a torrent client. When people see that much disk I/O or RAM usage they’ll probably switch to some other client instead. Why not keep the default to a low value so everyone is happy? If someone wants to allocate more RAM they could change it from settings.

This change came about because the vast majority of users have < 32 GiB RAM, and for them the previous fixed disk cache value was sometimes a major performance bottleneck.

[ghost]

Most people would not want to waste that much RAM on a torrent client. When people see that much disk I/O or RAM usage they’ll probably switch to some other client instead. Why not keep the default to a low value so everyone is happy? If someone wants to allocate more RAM they could change it from settings.

This change came about because the vast majority of users have < 32 GiB RAM, and for them the previous fixed disk cache value was sometimes a major performance bottleneck.

I see. But even on a 16/8 GiB system this would use an amount of RAM which is "significant" to some users. Specially when other alternatives might be using lesser amount of RAM with quite similar performance why would anyone opt to use qBit? Unless someone can prove that using a 500 MiB cache instead of 64 gives a significant performance boost, this will remain up to debate.

[FranciscoPombal]

I see. But even on a 16/8 GiB system this would use an amount of RAM which is "significant" to some users. Specially when other alternatives might be using lesser amount of RAM with quite similar performance why would anyone opt to use qBit? Unless someone can prove that using a 500 MiB cache instead of 64 gives a significant performance boost, this will remain up to debate.

I did not claim that 500 MiB is a better value than 64 MiB. My point is that currently, libtorrent assigns "reasonable" values for users that have 512 MiB RAM and up, with the exception of users with 2 GiB (for these users, auto sets the value to ~68 MiB, which is basically the same as before). Here, "reasonable" means "larger enough than 64 MiB to not be a bottleneck but not so big that users would complain about excessive RAM usage".

To quote my table above, extrapolated from the libtorrent code:

Some examples:

* 512 MiB RAM: 512/10 = ~51,2 MiB allocated for the disk cache (actually a bit smaller than the current default, which with this little RAM is probably a good thing)

* 1 GiB RAM: 1024/10 = ~100 MiB (only slightly bigger)

* 2 GiB RAM: 2048/30 = ~68 MiB (about on par)

* 4 GiB RAM: 4096/30 = ~136 MiB (bigger, not unreasonable)

* 8 GIB RAM: 8192/40 = ~200 MiB (bigger, not unreasonable)

* 16 GiB RAM: 16384/40 = ~400 MiB (bigger, not unreasonable)
  ...

It's with 32 GiB and up that the current libtorrent logic starts setting values that are perhaps too big. But like arvid mentioned, all this will be resolved in libtorrent 2.0. Nobody will have to fiddle with setting the amount of cache anymore. Until then, this is a workaround to:

• prevent tanking performance due to too little cache
• dissuade users from setting it to silly values like 1 GiB and up.

[ghost]

On a desktop environment 400 MiB could be unreasonable to many users. I believe the values that are automatically set by libtorrent should be re-calibrated to not go over a pre-set maximum. Like don't set anything beyond 200 MiB for even >8 GiB systems. The user always has the option to configure the cache settings if he's experiencing a bottleneck. Cache related bottlenecks will not happen in every use case. One guy could download a 400 MiB file to his cache but if no one is downloading from him, keeping that data in cache and wasting the memory sounds unreasonable to me. Also downloading to cache will not give any significant speed boost as well if the network is not fast enough. So is it ideal to use a large cache even if someone has a very slow network?

[arvidn]

it seems reasonable to make the "logarithm" extend past 4 GiB. I'm open to suggestions. Here's the current logic: https://github.com/arvidn/libtorrent/blob/d33b0506a0f4cd61860fd54864267c82d3e171e3/src/disk_buffer_pool.cpp#L294

Maybe there should be another tier at 8GiB

[FranciscoPombal]

@arvidn
I think these would be reasonable target values:

• 8 -> 200 MiB (same as now)
• 12 -> 250 MiB
• 16 -> 300 MiB
• 24 -> 350 MiB
• 32 -> 400 MiB
• >32 -> 512 MiB

I don't think it makes sense to fine tune too much for more than 32 GiB since:
a) it is already a huge amount
b) libtorrent 2.0 will probably come around way before systems with >32 GiB ram become common place among the majority of the user base (whereas systems with 8-16 GiB RAM are already common now).

@An0n666 thoughts?

[xavier2k6]

• 512 MiB RAM: 512/10 = ~51,2 MiB allocated for the disk cache (actually a bit smaller than the current default, which with this little RAM is probably a good thing)

• 1 GiB RAM: 1024/10 = ~100 MiB (only slightly bigger)

• 2 GiB RAM: 2048/30 = ~68 MiB (about on par)

• 4 GiB RAM: 4096/30 = ~136 MiB (bigger, not unreasonable)

• 8 GIB RAM: 8192/40 = ~200 MiB (bigger, not unreasonable)

• 16 GiB RAM: 16384/40 = ~400 MiB (bigger, not unreasonable)

I think the current method should be kept as is but with the addition of below:

Maybe there should be another tier at 8GiB

A new tier(s) at 8GiB/16GiB or 32GiB

There should probably be a MAX Upper Limit Cap for the manual setting of the disk cache setting too. (Just like the outstanding memory when checking torrents is capped at 1024MiB)

Should manual disk cache setting only have a range up to max 1024MiB??
this would fall under the current calculation:

• 32 GiB RAM: 32768/40 = ~800 MiB

[ghost]

it seems reasonable to make the "logarithm" extend past 4 GiB. I'm open to suggestions. Here's the current logic: https://github.com/arvidn/libtorrent/blob/d33b0506a0f4cd61860fd54864267c82d3e171e3/src/disk_buffer_pool.cpp#L294

Maybe there should be another tier at 8GiB

If this is the current logic then the above mentioned calculations are not correct.
You're take a 40th of everything exceeding 4 GiB
a 30th of everything exceeding 1 GiB
and a 10th of everything below a GiB and adding those up to get the final cache value?
So on a system with 8 GiB RAM it would be using around ~256MiB not ~200MiB?
Anyways, if that's the case then adding another tier on 8GiB would help.

[FranciscoPombal]

@xavier2k6 the table you quoted is what this PR implements. In the comment above yours, I suggested new cache values for systems with large amounts of RAM, which is what we are now talking about.

Without this PR, (i. e. in all stable releases of qBittorrent thus far), the cache is set to 64 MiB for everyone.

@An0n666
8 GiB RAM = 8192 MiB ; 8192/40 = 204,8

[ghost]

8 GiB RAM = 8192 MiB ; 8192/40 = 204,8

The code doesn't do a 8192/40. Check the link.

[xavier2k6]

the table you quoted is what this PR implements.

I thought the quoted table was what the "auto" option implements currently ie the libtorrent "logarithm"
If that's true then my opinion is the same, it should stay as is & tweaked with new tier(s) & I do apologise in advance if I've taken it up wrong/misunderstood.

In the comment above yours, I suggested new cache values for systems with large amounts of RAM, which is what we are now talking about.

You are suggesting to lower the cache values though on what the "logarithm" does currently??

I think these would be reasonable target values:

• 8 -> 200 MiB (same as now)
• 12 -> 250 MiB
• 16 -> 300 MiB
• 24 -> 350 MiB
• 32 -> 400 MiB

• 32 -> 512 MiB

BTW - I am a plus 1 for implementing the default of "auto" if it does in fact use the libtorrent "logarithm"