fasresume saving causes too much writes to disk

[reyaz006]

It seems like qBittorrent was the main reason why my SSD recently died. It was a good one, MLC-based. I've started noticing reallocated sectors few months ago, and somehow haven't realized that it may die so soon.
On my new SSD, judging by the data from the vendor tool, around 5 GB got written to it during 1 day.

So it all seems to come to concern I was talking about in #4315 (comment) - every few minutes qBittorrent saves data about current torrents, even if no download progress has been made. I'm not quite sure about amounts of data that was being rotated on my disk daily with each qBittorrent version, but right now on latest v4.1.1 it seems smaller - my very rough guess is about 2-5 GB per day. I think it was much bigger before - I remember that every single file was being re-written in BT_backup folder. My guess is that it was around 15 GB per day.

I'd like to discuss the following:

1. The ability to disable this kind of constant re-writing in BT_backup folder. Users should know that it's only being done for preserving statistics, for torrents that are already completed. So the reason of ~99% of total bytes written may be just about that - to keep various statistics.
2. The ability to choose mechanics of this process. Right now it's possible to only choose the interval. I may want to set the different target folder, to keep it away from my SSD.

Edit: It seems I was wrong about GB per day numbers, see my comments below.

[FranciscoPombal]

@reyaz006
What is the specific model of your SSD? How long have you had if for?
15 GB per day is about 5.5 TB/year, good SSDs such as 850 EVO have endurance ratings of >150 TB written, so even if qBit is writing that much data, it was likely not what killed your SSD.

Of course, if you download many, many large torrents, the data from the torrents themselves can certainly be significant.

[FranciscoPombal]

@reyaz006 However, you may have a point about the constant saving of fastresume files for completed torrents.
@Chocobo1 @glassez What does qBittorrent currently do? I am not knowledgeable about the current behaviour of saving fastresume files
Related:
#4704

[reyaz006]

What is the specific model of your SSD?

840 Pro. It lived less than 5 years, and most of this time I've used qBittorrent. I didn't even had a swap file on it, and I'm pretty sure I was careful enough to not overuse the TEMP folder on it. I don't recall any other application that would write that much data on it, except qBittorrent.

Of course, if you download many, many large torrents, the data from the torrents themselves can certainly be significant.

Even if I don't use SSD as a download location? I never did.

[FranciscoPombal]

Even if I don't use SSD as a download location?
Only if you use it as the download location. of course. If you never did, then that was not the problem.

I didn't even had a swap file on it, and I'm pretty sure I was careful enough to not overuse the TEMP folder on it.

You did not mention in the OP, but from this I assume you are on a Linux system? Can you provide more details, such as version, etc?

Meanwhile, you should wait for the reponse of the developers that I pinged in my previous post. I want to know more about fastresume saving as well (and I am not able to test/look through the code myself right now, too occupied with other stuff).

[reyaz006]

You did not mention in the OP, but from this I assume you are on a Linux system? Can you provide more details, such as version, etc?

Windows 8.1 x64. I also don't update it frequently.

[Seeker2]

Only the very latest libtorrent versions do block writes and reads under qBitTorrent running on Windows, which means they caused immense write amplification -- over 10 times the size of the torrents.

Older versions were doing 16 KB individual non-coalesced writes which meant SSDs had to do lots of block erases extremely often which is the hardest on them.
Newer versions will hopefully be able to write whole pieces in 1 go -- MUCH better, but still runs into problems with multi-file torrents:
https://qbforums.shiki.hu/index.php/topic,2627.msg12725.html#msg12725

[glassez]

qBittorrent periodically writes fastresume files (one file per torrent). Default interval is 3 minutes. You can configure it in Advanced Settings.

[reyaz006]

I'd like to correct myself - where I said about 5 GB during 1 day on a new SSD, it's actually 50 GB. I got confused with the number in the application - the difference was 0.05 TB.

so if you buy a 1TB SSD it will survive 4-6PB write without problems

That can't solve the problem with excessive writing of largely unnecessary data by qBittorrent. Also, I may not be able to afford such hardware anyway.

Newer versions will hopefully be able to write whole pieces in 1 go

I think you are not talking about "fastresume" data.

[Symbai]

@reyaz006 Workaround (yes I know it doesn't fix qbittorrent writing files at all) is to create a symbol link for qbittorrents folder where it writes the data to. https://superuser.com/questions/1020821/how-to-create-a-symbolic-link-on-windows-10 just link it to a folder on a different hard disk.

[FranciscoPombal]

@reyaz006

Newer versions will hopefully be able to write whole pieces in 1 go

I think you are not talking about "fastresume" data.

Indeed he is not. He is talking about the way qbittorrent writes the files from a torrent themselves to disk on Windows, and he has a very important point.
Based on what @Seeker2 and @glassez said, the .fastresume files are indeed most likely not the problem. They are too small and are written too infrequently (on default settings, 3 minutes) to pose a problem for modern SSDs, be they planar NAND or 3D NAND based.

The problem is much more likely to be the (lack of) write coalescing on Windows - which was only fixed in libtorrent 1.1.8. Unfortunately, there is still no official qBittorrent version built against libtorrent 1.1.8. You have to wait or build it yourself, or try one of the experimental builds posted in related issues.
Though I think in the current version the setting can still be enabled (checkbox in advanced settings).

Related issues: #9061 (the discussion focuses on performance, but the underlying problem is the same, the small 16 KiB writes).
Related: https://github.com/arvidn/libtorrent/releases/tag/libtorrent-1_1_8

[thalieht]

It's not impossible for qBt to write that many gigs per day. I for example have 111 torrents and my BT_backup folder is 4.76 MB. Assuming the default 3 min interval between saves that means about 2.2 GB per day, imagine someone with thousands torrents. Hmm i hadn't realized it was that much! Gonna try that symlink thing.

[Chocobo1]

Assuming the default 3 min interval between saves that means about 2.2 GB per day, imagine someone with thousands torrents.

Would it be reasonable to make the default 10 mins?
Using the same example, data writes goes down to ~690 MB per day.

[reyaz006]

symbol link

I've considered this before posting, and I stand by my proposition - to discuss the ability to review and change the mechanics, including changing the folder in settings. Resorting to symbol links is not really a good thing to suggest for normal users who only want to keep their SSD safe. Also it's a good idea to tell everyone about this issue because most people aren't even aware from what I understand.

Would it be reasonable to make the default 10 mins?

How about 120 minutes for default? And why not 1 minute or 30 seconds? I'd like to understand the logic behind those 3 minutes. If qBittorrent crashes, it'll have to recheck the files to proceed anyway (fastresume files are not needed for this). If it doesn't crash and gets shut down properly, it'll save those files anyway. Am I missing something?

[glassez]

I would make it less paranoidal by default (30 or even 60 min). At least for completed torrents.

[reyaz006]

So are there any downsides for case when I set it to 2^20 minutes?

[glassez]

If qBittorrent crashes, it'll have to recheck the files to proceed anyway (fastresume files are not needed for this).

There is also some statistics saved in fastresume files.

[glassez]

It would be nice to allow to disable intermediate fastresume saving, isn't it?

[reyaz006]

There is also some statistics saved in fastresume files.

Is there a good reason for this? "fastresume" seem to be about download progress. Statistics are something else. Wouldn't it make sense to use a separate db for statistics?

It would be nice to allow to disable intermediate fastresume saving, isn't it?

Unless I can understand what would that exactly mean for me, I can't decide.

[thalieht]

Unless I can understand what would that exactly mean for me, I can't decide.

I believe for completed torrents the saving interval is only useful for statistics. Everything else are saved on the spot when there is a need e.g. change savepath.

Well this issue certainly inspired me to greatly increase my saving interval.
There is also PR #7569
also linking relevant issue #5374

[Chocobo1]

It would be nice to allow to disable intermediate fastresume saving, isn't it?

Do you mean setting the save interval to 0?

[glassez]

Do you mean setting the save interval to 0?

Currently it won't work since it unconditionally translated to QTimer interval. So we should change it. Also it would be better to display "disabled" instead of 0 there.

[glassez]

Is there a good reason for this? "fastresume" seem to be about download progress.

It's the only way librorrent restore torrent state. Apparently it's time to think about some independent processing of this data.

[WolfganP]

As a workaround, is there any way to move the fastresume files to a ramdisk location? (ie any tmpfs mounted folder)

[zeule]

@WolfganP: https://wiki.archlinux.org/index.php/Firefox/Profile_on_RAM

[FranciscoPombal]

Didn't we come to the conclusion that the real issue was the write coalescing of the payload data? Or was I wrong there?

Also, IMO 3 minutes is no problem for active torrents, I would simply increase that value for completed torrents.

[Chocobo1]

Didn't we come to the conclusion that the real issue was the write coalescing of the payload data?

From thalieht's usage, it is hard to neglect its impact on SSD lifespan.
I guess there might be multiple reasons, hard to say which is the main one.

[reyaz006]

I've tried the following:

• Set the interval to 480 min, and see how much data would be written during 24h. The result is ~0.00 TB (as reported by the vendor tool).

• Set the interval to 15 min, and see how much data would be written during 24h. The result is ~0.03 TB.

Also, I think the following needs to be considered: Think about what would be the best way to count the interval. Are you sure you want it in minutes? Not in % of ratio or anything else? Some active torrents may be just stuck for weeks and months. Would it mean their meta data is still going to be rewritten many times even though there is no actual dl/ul progress?

[reyaz006]

Again I'd like to mention the following points.

1. The user needs to be informed about what exactly you are saving inside fastresume files. This way he can decide about the compromise if he needs it.

2. It may be useful to allow user to configure the folder where qBittorrent would save its data. Maybe a portable mode (keep everything in actual executable folder) would solve this.

3. If my guess about it is correct, the use case for disabling intermediate fastresume saving is probably when the user launches the application and keep it running only when his PC is working, e.g. he launches qBittorrent each morning and shutdown the PC each evening. If his PC is stable and he does not expect crashes or random power outages, it should work just fine - qBittorrent will keep the data in RAM, and will write it all to disk upon shutting down each evening. Correct me if there is something wrong here.

4. Again, since I don't know how exactly this works, I explore another assumption: the fastresume saving will still take effect even for stalled torrents, while no new data gets transferred for extended periods of time. If this is correct, is there a good reason to still use minutes for measuring saving intervals? Why not use the torrent progress %% and/or its sharing ratio?

[thalieht]

thalieht commented on Jul 3, 2018:

BT_backup folder is 4.76 MB

At this point it doesn't matter but i feel the need to say that i just discovered the .fastresume files are only a tenth of that size. The rest are from the .torrent files :)

[FranciscoPombal]

IMO this has been adequately addressed by #9161.