Disk Overload 100% on external exFAT 4K Advanced Format HDD (GPT)

[wtester7]

qBittorrent version and Operating System

Using qbittorrent 4.1.0 alpha from Chocobo1: #8295
on latest updated Windows 7 SP1 x64

What is the problem

If you're using an external formated exFAT 4K Advanced Format HDD (GPT) with qbitorrent 4.04 on Windows you will always end up with 100% disk overload.

I found a thread with a similar issue ( disk performance ) #8295
and was using @Chocobo1 , @arvidn test build qbittorrent 4.1.0 alpha from this thread and the disk performance was much better but there is still a problem.

If you add lots of new torrents into qbittorrent the disk overload will hit you in few seconds up to 100%.
As I saw qbittorrent allocates the files with its size to the external exFAT HDD and this will trigger the 100% write cache overload. After you wait ( depends on the size of the torrent and how many you add at the same time, could take from 10 minutes to 2 hours ) when the allocation and the reserved size of its torrents on the external exFAT HDD is finished, the disk overload will also disappear ( 0% write cache overload ) and you have again maximum download speed...

Settings were used:
Pre-allocate disk space for all files: off
Disk cache: -1 ( auto )
Disk expiry interval: 60 s
Enabled OS Cache ( doesn't matter if its on or off )

So I think the problem has to do with pre-allocation of the files, the option is disabled but qbittorrent is still pre-allocating the files, that's when the 100% disk overload occurs!

I hope you can fix this annoying issue, thank you!

P.S Does the constant 100% write cache overload over a longer period of time hurt or damaged the HDD? Or is the process of data just in queue and it doesn't matter or effect it? I am asking because the HDD is aggressively working non-stop...

[arvidn]

What does “disk overload” refer to specifically? Is there some metric?

If you end up having a lot of disk activity when adding a torrent, you may want to make sure you aren’t allocating files

[arvidn]

I don’t know any details about exFAT, but it doesn’t sound like a file system that supports sparese files, which means it would allocate space, and write zeros, up front regardless. Your only options are to use sequential download or use a real filesystem

[wtester7]

Yes it would seem that exFAT doesn't support sparse files, so the only option would be to USE the option: Pre-allocate disk space for all files: ON to prevent the 100% write cache overload right?

I formated the disk to exFAT with GPT because I didn't want to use NTFS due to filesize/partition limitation on Windows...

P.S Does the constant 100% write cache overload over a longer period of time hurt or damaged the HDD? Or is the process of data just in queue and it doesn't matter or effect it? I am asking because the HDD is aggressively working non-stop...

[arvidn]

Yes it would seem that exFAT doesn't support sparse files, so the only option would be to USE the option: Pre-allocate disk space for all files: ON to prevent the 100% write cache overload right?

no, that will also allocate the files up front. But if you combine it with increasing the number of disk I/O threads you may be less likely to stall other disk writes (from other downloading torrents) while allocating the files.

You can also not pre-allocate the files and enable sequential downloading, which will have a bias towards downloading pieces in-order, and hence allocate the file as you go.

[Keeperdy]

Same problem with 100% overload. External USB 2.0 HDD with 2 volumes - ExFAT/NTFS, Win 7 x86. One 2 Gb file download, average speed 9-10Mb/s, NTFS - all OK, exFAT - approximately 40-50% complete - 100% disk overloaded. Have already tried different settings with pre-allocation and cache size - nothing helped.

[Chocobo1]

Anyone here might want to try running qbt with administrator privileges (with Pre-allocate disk space for all files: ON). The file allocation should become instantaneous (at least with NTFS).

[arvidn]

ah right, I forgot about that. When running as administrator libtorrent has access to SetFileValidData(), which allocates space without zeroing it.

[Keeperdy]

I always run qbt with admin rights

[wtester7]

Well it's not much of a big deal, you just need to wait when the allocation of the files is completed,
then the overload is over, IF it dosen't somehow damage the HDD ( asked already two times ):

Does the constant 100% write cache overload over a longer period of time hurt or damaged the HDD?
Or is the process of data just in queue and it doesn't matter or effect it? I am asking because the HDD is aggressively working non-stop...

[Keeperdy]

Wait? What for? It's just wasting of time. Now I download a 30Gb file to NTFS volume (one hour) and then copy to exFAT volume (one hour). Tried to wait for exFAT 100% overloading finish - more than 2,5 hours! Nothing, just 2-3 Kb/s speed!

[wtester7]

Did you use Chocobo's test build 4.1.0 alpha from #8295 ?
Here is the direct link: https://github.com/Chocobo1/temp/raw/master/5/qbittorrent.exe

Because with this version the disk performance is better but the allocation of course still exists.
But somethings wrong with the description you described, because yesterday I tried one file with 25 GB and it takes about 5-10 Minutes to allocate and simultaneously multiple files with 500 GB about 2 hours, using an external USB 2.0 ( max about 25 MB/s ) exFAT HDD. After the allocation is finished there are no more problems, 0% write cache overload, maximum speed...

[Keeperdy]

No, but I'll try this test build. Yeah, that's correct - 2,5 hours and nothing changed. The main problem that even a single small file 2 Gb makes 100% overload to happen just being completed for 40-50% (more than minute). NTFS works perfectly - one HDD with two volumes, one cable, identical conditions!!! Now downloaded 53 Gb file to NTFS volume - no disk overloaded alert! That's weird!

[wtester7]

I think you have misunderstood my post.
With the Chocobo's test build 4.1.0 alpha , there are no more write cache overloads ( 0% ) while downloading. The 100% write cache overload occurs only when adding new downloads to qbittorrent due to forced pre-allocation of files with exFAT. When allocation of the new download files is finished there are no more write cache overloads! You have maximum download speed all the time and 0% write cache overloads, doesn't matter how many downloads are running at the same time...

[Keeperdy]

I had the same thing with uTorrent - 5-10 min for prellocation (100% disk overloaded) than OK - maximum speed, no problems! Changing some settings gave me no disk overload at any time, but it only worked for NTFS. Recently I made a second volume for exFAT and this overload problem began again, but now during downloading, not in the beginning like it was with uTorrent. I'll try alpha build and find out whether it will help solving this problem

[wtester7]

@arvidn and @Chocobo1 : How do you increase the disk I/O threads? I didn't see any option in the Advanced Options in qbittorrent...

And please answer this, I don't want to trash my HGST HDD, thanks:

Does the constant 100% write cache overload over a longer period of time hurt or damaged the HDD? Or is the process of data just in queue and it doesn't matter or effect it? I am asking because the HDD is aggressively working non-stop...

[Seeker2]

Yes, it's at least slightly hard on the disk:
#8630 (comment)

...And yes that will be vastly reduced by recent fixes in libtorrent that are yet to be committed to qBitTorrent.

[wtester7]

Hey Guys,

I did some tests yesterday and I found out that with the option:
Pre-allocate disk space for all files: ON

the allocation of files with the write cache overload seems much shorter! I added 10 downloads at the same time, the write cache overload 100% hits after some short time due to file allocation, but the completion of the file allocation feels much faster! When exFAT filesystem does not have sparse file support and it is forcing file allocation ( doesn't matter if you disabled it, it will just ignore it ) nonetheless, why not just enabled it :) , it seems qbittorrent/libtorrent is working better if it's ON.

Can anyone else also confirm? Thanks

[wtester7]

OK, confirmed!

Did another test and with Pre-allocate disk space for all files: ON
( with administrator privileges, using Admin account all the time ) and you have instantaneous file allocation without any write cache overload! Just saw the file sizes on the HDD for the downloads and the sizes were instantaneously reserved on the HDD.

Final summary:
Used qbittorent 4.1.0 alpha ( 64 Bit version ) with administrator privileges from:
https://github.com/Chocobo1/temp/raw/master/5/qbittorrent.exe

with these settings:
Pre-allocate disk space for all files: ON
Disk cache: -1 ( auto )
Disk expiry interval: 60 s
Enabled OS Cache

Problem solved 👍 , thanks @Chocobo1 , @arvidn and the other participants =)

[wtester7]

Also another important note, in previous test I added more than 10 torrents at the same time ( which could cause the write cache overload, but file allocation is still much faster with the pre-allocation option on! ).

In the latest test, I added just one torrent at a time, waited till download started then added another one and so on. With this method there was no write cache overload and instant file allocation while maintaining maximum download speed!

[Chocobo1]

Good! I'll close this issue then.
And you can grab the official v4.0.5 when it's released, that version will contain the fix too.