VFS Cache: control caching of files by size

[trumblejoe]

What is your current rclone version (output from rclone version)?

rclone v1.51.0-151-gfc663d98-beta

• os/arch: linux/amd64
• go version: go1.14.1

What problem are you are trying to solve?

File-level encryption w/ file-name encryption creates small, must-access files on a per-directory basis. These files will often be rate-limited by the underlying providers resulting in errors and temporary, potentially permanent, service bans. Additionally, this option below would help create more logical filtering rules for --vfs-cache-mode full given the fact cloud storage often greatly exceeds local storage.

How do you think rclone should be changed to solve that?

By implementing options for controlling the --vfs-cache-file-max-size and vfs-cache-file-min-size, you would be able to create a range of files which would be unaffected by vfs-cache-mode full. These unaffected files should default back to vfs cache-mode writes keeping in-line with the hierarchy structure.

Example File-Sizes :
25k, 1g
5g, 100g

File-max-size 26k.
Caches : 25k
Un-cached : 1g, 5g, 100g

File-min-size 2.5g
Caches : 5g, 100g
Un-cached : 25k, 1g

File-max-size 26k
File-min-size 6g
Caches : 25k, 100g
Un-cached : 1g,5g

Thoughts? Is the get file size function fast-enough to support these operations on-the-fly?

[ncw]

I think this could certainly be done.

I'm unsure how it would help in your scenario though? For --vfs-cache-mode full the small files are downloaded to disk. If you exclude them then the small files will be downloaded on the fly. Can you explain further?

[trumblejoe]

In my case, I'm not interested in excluding the small files actually, only in excluding larger files from monopolizing disk space. I actually don't run vfs-cache-mode full over writes because with my use-case my local storage fills up far too quickly with cache-mode full. I see this request as a way to make cache-mode full more useful than what it already is.

[darthShadow]

@ncw You did add this in https://forum.rclone.org/t/rclone-vfs-cache-maximum-size-per-file/30037/6 under the https://github.com/rclone/rclone/tree/fix-vfscache-strategy branch.

Would it be possible to add this to the mainline too? Any help that you require that's preventing it's merge?

[ncw]

I had forgotten about that code!

I rebased it here if anyone wants to try it

v1.61.0-beta.6571.6f53463c3.fix-vfscache-strategy on branch fix-vfscache-strategy (uploaded in 15-30 mins)

@darthShadow are you finding that code useful?

Would it be possible to add this to the mainline too? Any help that you require that's preventing it's merge?

According to the notes in the commit

• this needs docs, and maybe needs to be configurable.
• needs tests also

I think maybe its worth discussing the first point.

I was thinking of introducing a new parameter --vfs-cache-strategy which would have the following values

• lru - the current
• lru-sp what the patch implements which is "a size-adjusted and popularity-aware LRU replacement algorithm"

This could potentially have other values - see the wikipedia article for ideas. Most of these would only be a few lines of code to implement.

What do you think?

[darthShadow]

There was atleast one other person on the forum interested in it: https://forum.rclone.org/t/lru-sp-vfscache-strategy-roadmap/32919

Apart from that, I have pointed a few folks over on Discord to the linked branch for using that change and haven't heard any complaints yet ... 😉

I think we should merge it only with the additional lru-sp implementation for now and add the others as & when needed.

[a-earthperson]

+1 for interest in additional options for vfs cache modes. @ncw, having followed the related discussion on the forum, I'm keen on taking a crack at implementing an ARC type replacement policy. Perhaps I can start by adding documentation and tests for lru-sp.

[qcgzxw]

Looking forward to it~