Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rclone union mount allow more than one writeable remote #2737

Open
iamjrock opened this issue Nov 12, 2018 · 7 comments

Comments

@iamjrock
Copy link

commented Nov 12, 2018

Output of rclone version

v1.44, however this is a request for a new feature.

Describe the issue

This feature request has been discussed at length here.

rclone union mounts currently have the restriction that only the last remote can be written to (all other remotes are read-only). If we can remove this restriction then many rclone users who currently deploy unionfs or mergerfs could instead consolidate down to just using an rclone union mount. This is especially helpful for Windows/OSX users as there are no readily available union filesystem solutions like unionfs/mergerfs for these operating systems.

Whilst removing the read-only restriction would accommodate many scenarios, the very common use case that we would like to achieve is:

  • location-0 is a local temporary folder and has the highest read priority. location-1 is a cloud remote with most of our storage.
  • New files: When our local apps create new files, they are written to location-0 without any latency.
  • Moved/Renamed files:
    -- If the original file exists only on location-0: just mv within location-0
    -- If the original file exists only on location-1: just move (server-side) the file in location-1
    -- If the original filename exists in both locations: mv the file in both locations (server-side mv for location-1).
  • Updated files: If the file is opened and saved, or replaced with a file of a larger/smaller size (same filename):
    -- If the original file exists only in location-0: just overwrite the original file in location-0
    -- If the original file exists only in location-1: leave the old file in location-1 and write the new file to location-0
    -- If the original file exists in both locations: leave the old file in location-1 and overwrite the old file with the new file in location-0.
  • Deleted files: Delete from both locations.

This would then be cleaned up periodically with the following commands:

  • rclone move any file from location-0 to location-1 that is not identical (filename, subfolder, size) in both locations; then
  • delete any remaining files left over in location-0 (these are probably files that have the same filename and size in both remotes).

This periodic cleanup could be managed separately from this feature request, or included as union mount functionality.

The rclone forum thread also discusses various rclone.conf schemes that might be used to define the behaviour.

@BinsonBuzz

This comment has been minimized.

Copy link

commented Nov 12, 2018

Edit: Ignore below - just re-read @ncw comment in thread about letting rclone move remove the duplicate

Thanks for getting this going. A few questions/edits:

-- If the original file exists only in location-1: leave the old file in location-1 and write the new file to location-0
-- If the original file exists in both locations: leave the old file in location-1 and overwrite the old file with the new file in location-0.

Shouldn't that be delete the old file in location-1 and write new file to location-0? I'd rather avoid situations where I end up with duplicate files

Similarly:

rclone move any file from location-0 to location-1 that is not identical (filename, subfolder, size) in both locations; then

I'd rather it was move all files from location-0 (subject to any exclusions --exclude) including identical ones as I think rclone will just overwrite the old location-1 version, so I don't end up with duplicate files

@ncw ncw added this to the v1.46 milestone Nov 12, 2018

@iamjrock

This comment has been minimized.

Copy link
Author

commented Nov 12, 2018

I'd rather avoid situations where I end up with duplicate files

@BinsonBuzz, yes one of my earlier passes had that deletion, but I changed it after a comment from @ncw. It's a tradeoff:

  • Cleaner: Deleting from location-1 in these situations is "cleaner" in that there is no need to handle the overwrite of old versions in location-0 during the periodic move operation; vs
  • Safer: Not deleting is "safer" in that the old file is still in location-1 until it is explicitly replaced later. Like having a Trash can...

I went for safer. But perhaps this should be a configurable option?

including identical ones

I see your point. I guess I was thinking:

  • Less bandwidth is used if identical files are automatically excluded from the move; and
  • In my setup I can't always use an --exclude as I often don't know the name of the files in advance of the move.

@ncw ncw modified the milestones: v1.46, v1.47 Feb 9, 2019

@ncw ncw modified the milestones: v1.47, v1.48 Apr 15, 2019

@xiaolei0125

This comment has been minimized.

Copy link

commented Apr 22, 2019

@iamjrock Is there any new progress about unionfs?

@iamjrock

This comment has been minimized.

Copy link
Author

commented Apr 22, 2019

@xiaolei0125, nothing from my end. Sadly I haven't been able to help with rclone in the past few months. However I can see above that last week @ncw moved this enhancement request into the v1.48 milestone, so fingers crossed for v1.48!

@ncw ncw modified the milestones: v1.48, v1.49 Jun 19, 2019

@ncw ncw modified the milestones: v1.49, v1.50 Aug 27, 2019

@thestigma

This comment has been minimized.

Copy link

commented Sep 3, 2019

Just want to note my own interest in this. As a Windows user with few alternatives union seems like a fantastic tool but is a little too limited to be practical currently. Very much hoping we see progress on this soon.

@Inrego

This comment has been minimized.

Copy link

commented Sep 10, 2019

Indeed, same here. Windows user who doesn't really have any alternatives for this functionality.

@thestigma

This comment has been minimized.

Copy link

commented Sep 11, 2019

On NCW's request:
I will go through all the material tomorrow and make another post trying to refine all the previously discussed ideas + my own into a concrete "implementation recipe".
NO promises or time-frame at this point, so keep your expectations in check, but NCW has agreed to at least re-visit the topic.

[quote="thestigma, post:1, topic:11744"]
I think the existing issue and accompanying thread covered pretty much everything, but just as a short recap of what I think are the most critical missing features:

  • Instead of all inner remotes being read-only, they should all be writable but with the outer having write priority. (maybe have an option to mirror-write to all also, but this can be done with scripts so it's less critical)
  • Logic for deleting, moving, renaming ect. to work on all of the inner remotes also. Basically any operation on a file should apply to any drive that has that file - not just limited to the outer one.
  • Ability to set options-pr-remote (such as filter, or fast-list) and not just the end-point. Maybe this could be set directly into the Union remote config? If possible I think this would be a really clean way of doing it and you could likely keep full backwards compatibility with previous configuration formatting given that is currently so simple. This would be real useful for advanced setups, but depending on how much restructuring it would require I'd consider it a secondary priority compared to the two points above.
    [/quote]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.