mount: Make --vfs-cache-modes write queue files for upload

[ncw]

Currently when rclone is uploading files from rclone mount, it locks up windows explorer. This is because the file uploading blocks the directory because the file system operation is still in action in rclone and hasn't returned to the kernel and to the caller.

Proposed solution

• keep a queue of uploads
• when a file needs to be uploaded, add it to the queue and return immediately

Ideas

• rclone could limit the number of uploads
• rclone could cancel uploads for files which were modified again
• rclone could retry uploads
• rclone could persist the queue so uploads would resume when rclone was restarted - this would fix #FIXME
• rclone could just flush the local file in the FUSE flush call and then do the proper close in the FUSE release call, sure that uploads will be retried with the queue.

See: https://forum.rclone.org/t/rclone-mount-locking-up-file-explorer-windows/9921
See: https://forum.rclone.org/t/vfs-cache-mode-writes-upload-expected-behaviour/8014

[calisro]

What about implementing this with a flag like --async so people can choose the behavior depending on their case.

[ncw]

What about implementing this with a flag like --async so people can choose the behavior depending on their case.

That is not a bad plan. Probably a mode to disable it.

I chatted over the plan with Bill Z (the author of WinFSP who knows a lot more about filing systems than me) and he said that it was a pattern that he'd used before and it was a good one!

[maxim]

This would be a great addition 👍 because it's similar to "offline uploading" in cache remote, and cache is not being maintained quite as much anymore. This means VFS could get to the point where it can replace and deprecate cache. That said, there're a couple of suggestions I'd like to make.

1. Queue delay by time and size

One of the good things about the queue is that while files are in the queue they would still appear in mount and be stored locally. Keeping files local for a while is desirable for various use cases, and it would be great to give users some control over this behavior:

• A flag like --vfs-queue-wait-time=24h for delaying uploads by at least this long.
• A flag like --vfs-queue-size-limit=500G for uploading files only when queue crosses this total size.

2. Directories should be recognized

One of the problems with cache remote's offline uploading is that it doesn't track directories in the tmp_upload_path. We should avoid making the same mistake here.

[thestigma]

Yes! This would solve a lot. It would be fantastic if I can just hand over uploads to my home-server at maximum speeds of the HDD and network - and then rclone will take care of uploading it without needing anything from the sender or clogging up the OS. It's definitely something I've felt was missing (and the cache backends upload has a lot of issues).

I don't want to derail this too much, but consider making the VFS-temp-upload also function as a true write-cache, or at least design to make this possible later on. By that I mean keeping files after uploading (as long as max-size allows, then poll and remove oldest). Use those already uploaded files to instantly deliver any files that were recently uploaded and then re-downloaded shortly after (a fairly frequent use-case at least for me).

Alternatively make the primary VFS write-cache work like that (I assumed it did, but apparently not). Since it currently doesn't I assume there's some technical reason why it can't. That reason may not apply to a VFS-temp-upload folder? Just a thought.

[ErAzOr2k]

+1 This would be a huge improvement for rclone :)

[darthShadow]

Any chance that this may be possible for 1.49?

[thestigma]

A thought that may be relevant to this ...
To the best of my knowledge I don't think the VFS cache has any sort of persistence, and so if there is a crash, power failure or other other unexpected critical error I don't think the VFS cache will know to retry files that never made it to the cloud. (please tell me if my understanding on this is wrong).

This is already a risk of data-loss that should be mitigated as-is, but for now at least there has only been a handful of files in this state at a time. If a full async upload feature is implemented then I think it is very important that this hole is patched. It should be as easy as writing a "to upload" tasklist to file, or keeping not-yet-uploaded files in a separate folder that for the purposes of cache is viewed as merged.

[daniele-spl]

We would also appreciate this feature. It would be a really big improvement.

[daniele-spl]

Is there any possibility to speed up at least a beta for this?
Thanks

[eatnumber1]

One thing to consider to when designing is the (currently mostly unsupported) use case of applications which churn through lots of data on machines with little local disk. A torrent client is one such example.

Currently, you'll experience similar lock-up behavior as Windows explorer, but leveraging a feature that simply writes to local disk and moves on will be able to quickly fill the local disk. You need a mechanism to slow the writer.

One possibility would be to slow all write calls when the upload queue reaches a high watermark.

[mikn]

Is anyone working on implementing this?

[darthShadow]

This is already available in the latest beta builds. You can go through this thread for more info: https://forum.rclone.org/t/testing-for-new-vfs-cache-mode-features/16905

[ncw]

It is indeed! I'll close this issue now :-)

This will be released in v1.53 but you can find it in the latest beta now.

[hekmon]

Does it mean the cache backend will be discarded ? (as it is now superseded by the vfs integrated cache)

[ncw]

Does it mean the cache backend will be discarded ? (as it is now superseded by the vfs integrated cache)

In due course yes, but not immediately there is still more caching for the vfs to do.

[woshiniming007]

I tested now(1.55.0) and find that windows explorer would suspend for up to 2 minintes when I try to replace a large file being uploaded.

[darthShadow]

@woshiniming007 Please create a new forum post rather than necro-ing closed issues.