Increase `max_queued_disk_bytes` limit

[Obegg]

qBittorrent v4.6.0 - Windows x64 (lt20 qt6)
Qt: 6.4.3
Libtorrent: 2.0.9.0
Boost: 1.83.0
OpenSSL: 1.1.1w
zlib: 1.3

Edition	Windows 10 Enterprise LTSC
Version	21H2
Installed on	‎22/‎10/‎2023
OS build	19044.3570
Experience	Windows Feature Experience Pack 1000.19052.1000.0

There is a setting in qBittorrent called "Disk queue size" and from what I understand it's called max_queued_disk_bytes in libtorrent.
Currently the limit is 2147483647 / 1024 = 2097151 KiB = ~2 GiB as answered here.

My issue is apparently an uncommon issue where one of my hard drives from whatever reason read and writes from within qBittorrent is unstable, causing a total slow down in the entire download process in qBittorrent (yes, downloads in other drives will slow down to a halt), to clarify - it only happens with large amount of downloads, it doesn't happen when downloading a single file.
This issue appears only one out of the five hard drives I own, and a simple benchmark test using CrystalDiskMark shows the read and writes as advertised (above 100MBs), which I consider normal.

This specific hard drive is Transcend StoreJet 35T3 which I shucked (I did not tape anything to the conductors, I just formatted it once plugged in the machine) and the hard drive inside is ST3000DM007-1WY10G.
I've tried many solutions like: fresh install of Windows 10 with default qBittorrent settings, install qBittorrent with lt20 and non lt20 (yes, it happens with lt1.2), maximizing every setting inside qBittorrent to 2147483647, different SATA cable, enabling page file for this specific hard drive, disabling page file for this specific hard drive, test to RAM using memtest86 (pass 1 found 0 errors. I did not test pass 2, 3 and 4).
Checking using CrystalDiskInfo doesn't reveal any issues with any of the drives in the machine.

I have yet to know why this issue is specific to this hard drive, and trying to analyze this issue has become annoying, I have tried to open Resource Monitor at the time of the issue and found that the memory has many Hard Faults, which prompted me to try and enable/disable page file (because apparently that's how you fix it by simple Google Search).

To counter this issue (temporary or permanently) I suggest increasing the limit of max_queued_disk_bytes, I see it as a benefit from a couple a reasons, one being that RAM could be of use in machines with slow read or writes (without considering the reasons why), if a machine has 256GB of RAM there's currently no option to use all of it, second reason is that it will solve (as I said - temporary or permanently) my issue, third reason is that I don't see any reason why NOT to increase the limit, as I see it it will only be a good thing.

[arvidn]

2 GiB is quite a lot for this disk buffer. Are you confident that increasing it would improve your disk performance?
Are you sure you're not hitting your random access read rate limit?

It's not so easy to change this to a wider variable (int64) or to make it be counting 16 kiB blocks instead of bytes. The latter would be an incompatible change.

[Obegg]

Are you confident that increasing it would improve your disk performance?

I did not suggest this would improve disk performance, not at all.
I was suggesting increasing it because I did not want to suffer from slow download speeds when I reach 2GiB.
This would not improve disk performance.

Are you sure you're not hitting your random access read rate limit?

I have no idea, and I would love to know how I can check it in order to add more information about this issue to get it fixed.

This issue has been bothering me for quite a while when downloading many files all at once, so I would really appreciate it if you can guide me on what to do and steps to take in order to debug it properly for it to be fixed.

[arvidn]

so, you want to download into RAM quickly, and then (slowly) write it out to disk, yeah?
You won't be able to use the data while it's in the disk write queue though. It could probably pass hash checks and be available to upload to other peers.

That's probably the extent of the utility of keeping the data in the write queue over keeping it on the other peers (downloading at the speed the drive supports).

[Obegg]

so, you want to download into RAM quickly, and then (slowly) write it out to disk, yeah?

Yes, that's what I did want up till now, I no longer want that upon understanding your explanation.

So my question now is how do I solve that issue, what's the root cause of it and what's the solution?

[Obegg]

On second thought I would like to increase the max_queued_disk_bytes, 2GiB is bothersome.
Will you be able to do it?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Danny3]

On Linux (Debian (64 bit) the maximum Disk queue size, that I could set is: 2097151 KiB (1.999999 GiB).
I would like to increase it to a value somewhere between 4 - 8 GiB, since the computer I'm using has 32 GiB and not many things are running in the background.
The reason I want to increase it, is because I'm downloading on a SSD and I want to save its life as much as possible by lowering the number of unnecessary reads and writes.
And of course also for the case when I will be using a HDD for the downloads.
BTW, please also increase how fast the increase value (arrow up button) works as it takes a very long time to reach the maximum value.
And please make the value in MiB instead of the current KiB.
Thank you!

[PeTeFIN]

Hard to understand why this much more requested feature. On uTorrent with windows, i have it set to 48GB. Using a 1GB line, downloading a file, its literally impossible the spinning HD to keep up, the totally backed up HD will cause lag to the whole system.

The download will be going as fast as possible, while HD creates the file, lets say 100GB and starts writing into it. Both are done roughly the same time, since when the HD actually starts writing, it can usually outpace the download speeds, if you let it do its thing in peace and not overload it.

Maybe you could get around the problem by adding multiple ques, if the 2GB is hardcoded.

[Danny3]

I don't have enough free space on my laptop for downloads too, so I moved them to an USB stick.
Unfortunately, because the USB stick is pretty slow to both reads and writes, now doing other things on it it's insanely slow adn with checking torrents it's even slower.
The download and upload speeds are also not great, even though they should be considering I have more than enough RAM.
16 GiB and some of it it's compressed with ZRAM.
It's pretty unbelievable that on μTorrent I could set pretty high disk caches almost 2 decades ago and in Qbittorrent I cannot after so much time.
It also defeats the main purpose of RAM, which is to be used a a cache to speed up operations.
Having bought more than enough RAM is pointless if it's not used. Money and electricity is waste if not used properly.

Please remove these artificial limits or if you still insist into hardcoding them, please set them higher, like 8-16 GiB!
We need high speed downloads and uploads, but also be possible to work with other folders and files on those destination drives or somewhere else without the interface freezing or hiccuping / stuttering.
The storage size, and the internet speeds have improved a lot in the past decade, but the reading and writing speeds from and to hard disks and USB flash drives not so much.
Luckily RAM sizes also improved a lot.

Only things that remain is for them to be better used.

[Danny3]

Also, if you don't mind, please also change the counting units from bytes to MegaBytes or more specifically to MebiBytes.
As it's take too much time to increase it or decrease it with those spinners when they count in bytes.
I always have to do a GiB or MiB conversion in DDG to just copy and paste the value that I want, which is time wasting.
Hitting also lower hardcoded limits than expected also even more time wasting.