system.file.allocate.set=1 doesn't allocate space for all the files at start

[chros73]

I tried to put system.file.allocate.set=1 into my config, but the result is the same: it doesn't allocate space all at once in the beginning.

I found couple of posts/articles about this (like this), and they say that you have to recompile libtorrent with --with-posix-fallocate flag.
Is it still the case?

system: Ubuntu, fs: ext4, rtorrent-ps 0.9.6 / libtorrent 0.13.6.

Thanks

[chros73]

How shall I replicate the same effect (what linux command with which switch shall I use) that rtorrent uses to allocate files?

[chros73]

I rebuilt libtorrent with "--with-posix-fallocate" flag (and rtorrent 0.9.6 as well), but the result is the same as it was previously.

It turned out after a short test using a multi file torrent:

• it allocates All the space for those files only (!) that are started to written to, but not the rest

E.g. a 50 GB torrent occupies only e.g. 24 GB in the beginning of downloading.

And this issue is a sibling of this one: #202 .

[pyroscope]

If https://relict.blogspot.de/2013/06/rtorrent-new-file-size.html works, it should be done right as a PR and react to the setting's value.

[chros73]

it should be done right as a PR and react to the setting's value.

Good idea, the only question is how it should work! :)

A. Current state:

1. On Linux with supported FS (like btrfs, ext4, ocfs2, xfs) fallocate is used by rtrorrent by default. That's why --with-posix-fallocate shouldn't be used, actually it doesn't matter whether it's used or not.

2. As it was suggested on Files still occupy space despite being set to priority 'off'  #202 , fallocate is always used but ONLY for files that have been written to! The rest of the files is still untouched (size is 0).
   Btw, is there any reason why it is always on on those system? E.g. is there any performance issue, etc?

3. system.file.allocate directive doesn't do anything on these systems (look above), it's like it was always set to 1.

4. As it seems this working state isn't good for anything:
   a) we can't switch off file allocation
   b) we can't set file allocation at start time for ALL the files of a torrent

The result is a mixed bag. :)

B. Possible solution:

1. The suggestion on Files still occupy space despite being set to priority 'off'  #202 is working for switching it off, though I tried with changing this line instead to:
   static const int flag_fallocate = 0;

C. Questions:

1. what should system.file.allocate directive mean?

• if it's set to: 0 (switch off fallocate completely)
• if it's set to: 1 (use fallocate for all the files if it's available)

2. I can probably create a fix for A.4.a. (means ticket Files still occupy space despite being set to priority 'off'  #202) but not for A.4.b (this ticket).
   If it's still fine, I'll create a pull request for that, if we agree how should it work.

[kannibalox]

Regarding the file allocation of unwritten-to files, there are lot of
people who ( for better or worse) turn off certain files to save space.
Pre-allocating space for those would probably break expectations.

[chros73]

Since I've created a fix for A.4.a. (means issue #202), I've edited the title of this issue.

So, the remaining problem is:

• system.file.allocate.set=1 doesn't allocate space for ALL the files at start (some file size is 0), just for those files only (!) that are started to written to

[pyroscope]

f.multicall=,f.set_create_queued=0,f.set_resize_queued=0

try that in your watch schedule as part of the load commands.

[chros73]

Thanks, I'll try it out!

[chros73]

I've tried it out, but doesn't do anything:

• f.multicall=*,print=$f.is_create_queued= it's 0 for all the files by default (also without f.set_create_queued=0)
• f.multicall=*,print=$f.is_resize_queued= it's 1 for files that hasn't been resized yet (even with f.set_resize_queued=0)

[chros73]

I need help with this part (as I thought).

What we want is:

• if the config entry is 1 and when a download is started (but not when it's opened) then all the files should be allocated

Current logic:

• Download::open looks good (it sets all the necessary flags)
• Download::start looks interesting:
  • "our new behavior is to create all zero-length files with flag_create_queued set"
  • where does this happen?
  • I've tried simply change this to file_list()->open(flags);, but the result is the same
• FileList::open also looks OK.
• File::prepare responsible for calling the actual file allocation
• this is where File::prepare is called: FileList::create_chunk_part (if I'm right)

[chros73]

I've found it where it happens: FileList::open_file
If I change it to return node->prepare(MemoryChunk::prot_write, 0); all the files are properly allocated at start time but not when torrent is loaded (e.g. via load.normal), so that's perfect.

What left is:

• add a check for the config variable for it
• I don't want to allocate space for files that are off during load time, even if the config variable is set (allowing deselect files during load time, so they never occupy more space than it's necessary)

[chros73]

I've created a fix for this as well in libtorrent: rakshasa/libtorrent#109

[chros73]

Wiki page is updated.

[Firerouge]

What is the status of the fix? Compiled rtorrent seems to still fail at allocating all files at start, unless a chunk has first been downloaded from the file. In other words, files aren't fully preallocated upon adding a torrent.

[chros73]

That's normal, since the above linked fix hasn't been merged. Take a look at its comment.
If you want you can use it, but you need a proper queue management otherwise you will crash rtorrent easily.
Maybe I will commit what I managed to do so far (it's far from complete).

[adamwinn]

What's the status of this fix? I'm interested in system.file.allocate.set=0 working

[chros73]

What's the status of this fix?

Although it's only a partial fix but hopefully it will be merged soon.

I'm interested in system.file.allocate.set=0 working

That fix was merged into v0.9.7 .

[adamwinn]

That fix was merged into v0.9.7

@chros73 Thank you!