File pre-allocation on Windows without admin privileges (SetFileValidData)

[libTorrentUser]

There is just one problem with the current SetFileValidData solution. As you already know, the user may not have the required privileges for the call. When that happens the call will fail (as the code already expects) but nothing else is done.

The result, on Windows, is a useless sparse file. The user will think libTorrent is pre-allocating the file because he asked it do it, a not so tech savvy user will think the file was pre-allocated because windows explorer will "lie" and show the make believe size and the original problem pre-allocation was supposed to solve (disk fragmentation) will still happen.

I suggest falling back to the classic SetFilePointer,SetEndOfFile combo when that happens (when the user has no permission to call SetFileValidData). Yes, it will go back to the write zeros situation, but at least it will be pre-allocating instead of lying to the users.

[arvidn]

Are sparse files on windows useless? (excuse my ignorance, I haven't used windows in a long time).

I would expect non tech savvy users to just suffer through windows being slow, not attributing it to files not being pre-allocated.

I imagine you're saying this is a problem on NTFS, right? FAT32 will just not do sparse files at all, but I suppose it could still be somewhat fragmented until a late piece arrives and the bulk of the file is allocated.

Which branch are you referring to? RC_1_1, RC_1_2 or master?

afaict, all three branches do SetFilePointerEx(), SetEndOfFile() and then the (best effort) SetFileValidData(). What are you suggesting that's different from this?

My understanding is that the file won't be made sparse if the storage mode is set to allocate, isn't it enough to SetEndOfFile on a non-sparse file to get it to be pre-allocated?

[libTorrentUser]

The problem is happening on 1.1.11, the one supposedly used by qBittorrent 4.1.5. So I would say RC_1_1. All this running on Windows 7 64 on what I believe is a regular guest account.

If libTorrent already does what you said, which is pretty much what I wanted it to do, then either something is failing when the user does not have elevated privileges, or sparse files really messes things up. Or, of course, the qBittorrent guys did something weird in their build. Because looking at their code they are definitely setting the storage mode to allocate and it is resulting in a sparse file.

I didn't notice the problem on 1.2, but then again, I only used that one while doing some tests using Visual Studio, meaning I was on an elevated account. As soon as I get home I will try to check and see if I can reproduce it using 1.2.

And my understanding is that your understanding is correct :). The call to SetEndOfFile on a non-sparse file should be enough to pre-allocate it. At least it is how I have been doing for the last 10+ years. But I didn't know about SetFileValidData, though. It sounds very useful, but I don't know if something weird is going on because of it. It shouldn't, because my user should not have permission to make that call.

Regarding sparse files in general, on Windows on a NTFS HDD, I would say they are pretty much useless for torrents. Unless, of course, your torrent data is just a bunch of zeros. When you start filling the sparse file with valid data it will fragment just like a regular file (in case of simultaneous writes to other files on the same disk). If it was up to me, I would recommend removing that DeviceIoControl( FSCTL_SET_SPARSE ) call all together. Let your user choose between real pre-allocation, or no pre-allocation at all.

I'll try to get back to you later and tell you if I was able to reproduce it using 1.2 too.

By the way, why are there 3 (well, 2) different packages? What is the difference between libtorrent-rasterbar-1.2.0.tar.gz and the "non-rasterbar" version?

[libTorrentUser]

I can confirm the problem DOES NOT happen with 1.2.

Just tested it running this tutorial code under that same under-privileged user. The only changes I made were to add 2lines to force the desired allocation mode and sequential download.
atp.storage_mode = lt::storage_mode_t::storage_mode_allocate;
atp.flags |= lt::add_torrent_params::flag_sequential_download;

Works just as expected. Either it only happens with 1.1.11 or it is a qBittorrent only bug. Either way, since it certainly does not happen on the latest version, it is not libTorrent fault.

I am sorry for wasting your time.

[arvidn]

thanks for testing!

The reason I think this would work in 1.1.11 as well is that this code is virtually unchanged:

RC_1_1: https://github.com/arvidn/libtorrent/blob/RC_1_1/src/file.cpp#L2078
RC_1_2: https://github.com/arvidn/libtorrent/blob/RC_1_2/src/file.cpp#L1125
master: https://github.com/arvidn/libtorrent/blob/master/src/file.cpp#L711