Intensive IO read when seeding

[maisun]

Hi,
I switched from rTorrent to qBittorrent due to the fact that rTorrent reads the HDD intensively during seeding - in my setup rTorrent constantly read 50MB/s while uploading only 2-3MB/s.
Among many other good things, I found qBittorrent is better in that regard, but the issue still exists. I found in Netdata that qBittorrent reads between 5-15MB/s while seeding 1-2MB/s, I'd like to get down to the problem and hopefully something can be tuned with settings.
For reference I'm on qBittorrent v4.3.3.1 with linuxserver/qbittorrent docker image. I have 1Gbps up/down fiber connection. Below are the settings I use:

Any suggestion is appreciated!

[The5kull]

My guess it has something to do with both caches set rather high.
Try with "Outstanding memory.." on 32MiB and "Disk cache" on 512MiB.

[maisun]

My guess it has something to do with both caches set rather high.
Try with "Outstanding memory.." on 32MiB and "Disk cache" on 512MiB.

Thanks tried that but same problem :-(

[The5kull]

I forgot to ask, what kind of storage unit are we talking about? Your information is missing that.

[maisun]

I forgot to ask, what kind of storage unit are we talking about? Your information is missing that.

Sorry not sure what do you mean by storage unit?

[ArcticGems]

I forgot to ask, what kind of storage unit are we talking about? Your information is missing that.

Sorry not sure what do you mean by storage unit?

If it's internal/external and mechanical SATA/SATA SSD/M.2 NVMe etc...

[maisun]

The docker container runs on SATA SSD which is the system volume of my NAS. Downloaded files are on HDD Toshiba MG08 16TB. High Read io is on my Toshiba HDD.

[The5kull]

The docker container runs on SATA SSD which is the system volume of my NAS. Downloaded files are on HDD Toshiba MG08 16TB. High Read io is on my Toshiba HDD.

The only reason for the intensive IO read would be fragmentation if you use such a big capacity drive. Also make sure you have checked "Pre-allocate disk space for all files" if you use a mechanical drive.

[maisun]

The docker container runs on SATA SSD which is the system volume of my NAS. Downloaded files are on HDD Toshiba MG08 16TB. High Read io is on my Toshiba HDD.

The only reason for the intensive IO read would be fragmentation if you use such a big capacity drive. Also make sure you have checked "Pre-allocate disk space for all files" if you use a mechanical drive.

Ok, I have enabled the settings, and all files that I’m seeding were copied from another disk, so I don’t think it has something to do with disk fragmentation. Anyways the read io is constant, it’s not like reading for x minute then stop for y minutes.

[FranciscoPombal]

@arvidn ideas?

[arvidn]

is this using libtorrent 1.2.x?
If so, the read_cache_line_size setting could be lowered.

If this is using libtorrent 2.0.x, the read-ahead is a kernel setting.

[maisun]

is this using libtorrent 1.2.x?
If so, the read_cache_line_size setting could be lowered.

If this is using libtorrent 2.0.x, the read-ahead is a kernel setting.

I believe it’s Libtorrent V1.2.13.0.
Do you know how to change that setting in qbittorrent?

[stickz]

is this using libtorrent 1.2.x?
If so, the read_cache_line_size setting could be lowered.
If this is using libtorrent 2.0.x, the read-ahead is a kernel setting.

I believe it’s Libtorrent V1.2.13.0.
Do you know how to change that setting in qbittorrent?

You can't change that without building a new custom docker image. rTorrent is actually better for reducing random disk reads than qBittorrent. You most likely didn't configure the .rtorrent.rc file properly.

These settings will greatly reduce the amount of random disk reads/writes on rTorrent.

network.send_buffer.size.set = 32M
network.receive_buffer.size.set = 32M

[maisun]

is this using libtorrent 1.2.x?
If so, the read_cache_line_size setting could be lowered.
If this is using libtorrent 2.0.x, the read-ahead is a kernel setting.

I believe it’s Libtorrent V1.2.13.0.
Do you know how to change that setting in qbittorrent?

You can't change that without building a new custom docker image. rTorrent is actually better for reducing random disk reads than qBittorrent. You most likely didn't configure the .rtorrent.rc file properly.

These settings will greatly reduce the amount of random disk reads/writes on rTorrent.

network.send_buffer.size.set = 32M
network.receive_buffer.size.set = 32M

I actually tried the send receive buffer with rtorrent but it didn’t help. With qBittorrent the read amplification is about 3-5 times but with rtorrent it was 10-20 times

[T2JOESl4m2ZpNC]

The docker container runs on SATA SSD which is the system volume of my NAS. Downloaded files are on HDD Toshiba MG08 16TB. High Read io is on my Toshiba HDD.

The only reason for the intensive IO read would be fragmentation if you use such a big capacity drive. Also make sure you have checked "Pre-allocate disk space for all files" if you use a mechanical drive.

Ok, I have enabled the settings, and all files that I’m seeding were copied from another disk, so I don’t think it has something to do with disk fragmentation. Anyways the read io is constant, it’s not like reading for x minute then stop for y minutes.

I also believe to be having the same issue here qbittorrent v4.4.1 arch linux, Maybe interestingly this also happened when i moved a lot of files (500G) from a 1TB 2.5 HDD (SATA) to a new 1TB 2.5 HDD connected with a SATA to USB3.0 external case.
I moved the files using qbittorrent's Set Location feature.
i now see that 30-40 Mb/s constantly being read while only seeding around 200Kb/s and overall decrease in upload speed.
this also affected my download speed as now i have to pause everything that is seeding to download at full speed.

[terrellgf]

+1
I had the same problem in 4.4.x（4.4.0/4.4.1/4.4.2）

[DaMatrix]

cam confirm, this is definitely still an issue. i regularly see 10-30x read amplification when seeding.

this is on qBittorrent v4.4.3.1, with the data stored on an 8-drive btrfs raid1. all the files have been properly defragmented, so fragmentation is not the issue.

i checked the OS page cache using vmtouch on one of the larger torrent files i'm seeding, and got the following:

this was done 2 minutes after manually dropping the entire page cache (with echo 1 | sudo tee /proc/sys/vm/drop_caches). i watched the web UI the entire time: in that time, the torrent was active seeding for only about 45 seconds, during which barely 30 megabytes were uploaded to one client. why on earth is it necessary to access 1 gigabyte of (unique!) data for such a small uploaded amount?

[nosirrahdrof]

Anyone have any further information or discoveries from this problem? I seem to be suffering with this same issue.

[nosirrahdrof]

I've actually had to download a bandwidth limiter to limit the download bandwidth of the NIC i'm using. Was seeing 50 MBps and even higher consistently of read activity on my nas when seeding at around 1 MBps.

The network limiting software seems to work fine and seeding is proceeding as normal with no more read amplification. I'm not sure if this is going to cause problems long term or not, but I don't understand why Qbit was needing to read TB's per day while only uploading a tiny fraction of that.

If anyone has any discoveries as to why this is happening, I would love to hear about it.

[agneevX]

This is unfortunately still an issue with the latest version of linuxserver/qbittorrent, as of this post.

~160GB of read for 1GB upload.

---

[alexander-yakushev]

Chiming in. I started observing 6-8x read amplification when seeding with linuxserver/qBittorrent 4.5.2 when I switched my hard drives to BTRFS. Previously, I used it with ext4, and the disk I/O strictly matched the upload speed. I can provide any extra data if requested.

[lukefor]

I'm also seeing up to around 10x read amplification since moving from Transmission. Using XFS via NFS, also seen same behaviour with XFS directly. Kernel/NFS settings for readahead are all much lower than torrent piece size. Versions of qB/libtorrent are as shipped in Debian 12

[flisk]

i was seeing this on qbittorrent v4.5.2 after upgrading to debian bookworm, and it got a lot better after migrating my torrents disk from btrfs to xfs. read amplification is down from a factor of up to 15x to about 2-4x. it seems like btrfs doesn't play nice with high volumes of small random reads.