rclone mount - backends without support for DirMove and Move do not fall back to copy or upload / download #2539

tcf909 · 2018-09-05T22:51:25Z

My initial request was to support this feature, but it appears that rclone "should" already support this feature (falling back when a Move or DirMove is requested).

Unfortunately, this does not work as documented.

See below for details.

tcf909 · 2018-09-05T22:55:42Z

I noticed on the website, this is the stated method of execution, but it does not actually work on google cloud storage.

rclone mount --allow-other --fast-list --dir-cache-time 15s --cache-dir /shared/.cache --vfs-cache-mode full --vfs-cache-max-age 24h --vfs-cache-poll-interval 5m --low-level-retries 9999 $BACKEND $DIR

rclone v1.43

os/arch: darwin/amd64
go version: go1.11

root@www-0:/shared/sites/digitalspaces.app# mkdir test
root@www-0:/shared/sites/digitalspaces.app# mv test test2
mv: cannot move 'test' to 'test2': Input/output error

2018/09/05 22:54:27 DEBUG : /: Attr:
2018/09/05 22:54:27 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2018/09/05 22:54:30 DEBUG : /: Lookup: name="test"
2018/09/05 22:54:30 DEBUG : : Re-reading directory (1m12.131528408s old)
2018/09/05 22:54:31 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2018/09/05 22:54:31 DEBUG : /: Mkdir: name="test"
2018/09/05 22:54:31 DEBUG : /: >Mkdir: node=test/, err=<nil>
2018/09/05 22:54:31 DEBUG : test/: Attr:
2018/09/05 22:54:31 DEBUG : test/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2018/09/05 22:54:34 DEBUG : /: Lookup: name="test2"
2018/09/05 22:54:34 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2018/09/05 22:54:34 DEBUG : test/: Attr:
2018/09/05 22:54:34 DEBUG : test/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2018/09/05 22:54:34 DEBUG : /: Lookup: name="test2"
2018/09/05 22:54:34 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2018/09/05 22:54:34 DEBUG : /: Lookup: name="test2"
2018/09/05 22:54:34 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2018/09/05 22:54:34 DEBUG : /: Rename: oldName="test", newName="test2", newDir=/
2018/09/05 22:54:34 ERROR : test: Dir.Rename error: Fs "Storage bucket digital-spaces-www-local" can't rename directories (no DirMove)
2018/09/05 22:54:34 DEBUG : /: >Rename: err=Fs "Storage bucket digital-spaces-www-local" can't rename directories (no DirMove)

tcf909 · 2018-09-05T23:19:14Z

Also tried without any cache options -- same result (failure).

brechmos · 2018-09-08T02:15:56Z

I think I am getting a similar error (with a dropbox mount). I am moving one file on another that already exists.

$ mv nonce.tmp nonce.tmp2
2018/09/07 22:10:25 ERROR : nonce.tmp: File.Rename error: move failed: to/conflict/file/... 
2018/09/07 22:10:25 ERROR : nonce.tmp: Dir.Rename error: move failed: to/conflict/file/... 
mv: cannot move 'nonce.tmp' to 'nonce.tmp2': Input/output error

My mount options are uid=1000,gid=1000,allow-other,default-permissions,max-read-ahead=16M,_netdev (using the rclonefs script from the wiki).

Actually, I think my issue is #1965

rclone v1.43
- os/arch: linux/arm
- go version: go1.11

ncw · 2018-09-11T09:39:21Z

I think this issue is a duplicate of #1965 which is scheduled to be done soon.

@brechmos I think renaming should work on dropbox - do you get that problem every time?

brechmos · 2018-09-11T12:02:59Z

Sorry, yeah, I updated my comment above noting the #1965 connection but it was after I first posted :/

I just tried with dropbox and I am getting an error.

This is the mounting:

$ ~/bin/rclone mount -v --debug-fuse  Dropbox: /tmp/db
2018/09/11 07:44:27 INFO  : Dropbox root '': poll-interval is not supported by this remote
2018/09/11 07:44:37 ERROR : Test/a: File.Rename error: move failed: to/conflict/file/..
2018/09/11 07:44:37 ERROR : Test/a: Dir.Rename error: move failed: to/conflict/file/..

The error above came when I did:

$ cd /tmp/db
$ touch a b  # Make two empty files.
$ mv a b
mv: cannot move 'a' to 'b': Input/output error

I am quite new at rclone, so there might be some helpful other flags but with the reasonably bare mount command I am seeing an error.

Oh... so, interesting. I just did the same thing with Mega and it worked:

$ ~/bin/rclone mount -v --debug-fuse  Mega: /tmp/db
2018/09/11 07:48:42 INFO  : mega root '': poll-interval is not supported by this remote
2018/09/11 07:48:56 INFO  : test/a: Copied (new)
2018/09/11 07:48:57 INFO  : test/b: Copied (new)

It also works with pCloud:

$ ~/bin/rclone mount -v --debug-fuse pCloud: /tmp/db
2018/09/11 07:50:05 INFO  : pcloud root '': poll-interval is not supported by this remote
2018/09/11 07:50:23 INFO  : test/a: Copied (new)
2018/09/11 07:50:23 INFO  : test/b: Copied (new)

It also worked with Google Drive.

Tried Box and it did not work.

Huh, so it seems to be only Box and Dropbox (of the ones I tested) that are giving me issues.

All of the above is based on:

rclone v1.43
- os/arch: linux/arm
- go version: go1.11

If you want me to test a PR, I am happy to do it.

brechmos · 2018-09-11T12:30:23Z

I tried a few more times in different incantations and it seems to be ~~pretty~~ consistent (for me).

ncw · 2018-09-11T20:39:45Z

@brechmos great testing.

Can you put your findings in a new issue please - I think your problem is unrelated to this issue.

It seems to be that backends are inconsistent about whether they allow a file to be copied over or not.

tcf909 · 2018-09-11T21:26:57Z

@ncw your right does look like a dupe. The difference is this is Google Cloud Storage. Not sure how your tracking things. (Per feature, per backend, etc.) Feel free to close.

Looking forward to this being done. Will make using mount much more like a real fs for those backends that don't support move.

brechmos · 2018-09-11T21:44:53Z

Per @ncw I created a ticket for my findings #2569

ncw · 2018-09-29T14:11:03Z

I think this issue is now fixed with Move falling back to Copy if not availabe.

DirMove does not fall back to Move or Copy yet though. I'm uncertain as to whether that is a good idea or not as it might involve renaming 1000s of files for one directory rename.

tcf909 · 2018-09-29T15:34:58Z

To make mount operate more like a real dir wouldn't we need to make sure dirmove failed back? At least make it a flag like vfs so you can force a more filesystem type experience?

…

On Sat, Sep 29, 2018, 7:11 AM Nick Craig-Wood ***@***.***> wrote: I think this issue is now fixed with Move falling back to Copy if not availabe. DirMove does not fall back to Move or Copy yet though. I'm uncertain as to whether that is a good idea or not as it might involve renaming 1000s of files for one directory rename. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2539 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABfZI6Gy6wpbRnJJp-uxi_5Xmn-i1DHCks5uf3-AgaJpZM4Wb0hd> .

ncw · 2018-09-30T08:36:04Z

@tcf909 wrote:

To make mount operate more like a real dir wouldn't we need to make sure dirmove failed back? At least make it a flag like vfs so you can force a more filesystem type experience?

I guess it is a matter of tradeoffs...

To rename a directory rclone will

Call DirMove if it is implemented (see overview table for which backends do implement it)

works for 12/21 of backends

Call Move on each file if available

works for another 0/21 backends (all the backends which implement Move implement DirMove)

Call Copy then Delete on each file if available

works for another 6/21 backends

Download then upload each file

works for the remaining 2/21 backends (b2 & yandex)

At the moment rclone mount will stop at step 1) above. I think it could go down to step 3) possibly, but step 4) I think is too much, at least without a flag.

To put those into perspective let's say we have a directory with 1000 files and 100GB of data in. This is what rclone does for each of those steps

Do one API call
Do 1000 API calls
Do 2000 API calls
Do 3000 API calls, download 100GB and upload 100GB of data

What do you think rclone should do?

tcf909 · 2018-09-30T16:58:00Z

I agree with your analysis. The final step is too much without a flag. If you were able to cover the first three, and cover 22 of the 24 backends that would be great.

…

On Sun, Sep 30, 2018, 1:36 AM Nick Craig-Wood ***@***.***> wrote: @tcf909 <https://github.com/tcf909> wrote: To make mount operate more like a real dir wouldn't we need to make sure dirmove failed back? At least make it a flag like vfs so you can force a more filesystem type experience? I guess it is a matter of tradeoffs... To rename a directory rclone will 1. Call DirMove if it is implemented (see overview table <https://rclone.org/overview/#optional-features> for which backends do implement it) - works for 12/21 of backends 1. Call Move on each file if available - works for another 0/21 backends (all the backends which implement Move implement DirMove) 1. Call Copy then Delete on each file if available - works for another 6/21 backends 1. Download then upload each file - works for the remaining 2/21 backends (b2 & yandex) At the moment rclone mount will stop at step 1) above. I think it could go down to step 3) possibly, but step 4) I think is too much, at least without a flag. To put those into perspective let's say we have a directory with 1000 files and 100GB of data in. This is what rclone does for each of those steps 1. Do one API call 2. Do 1000 API calls 3. Do 2000 API calls 4. Do 3000 API calls, download 100GB and upload 100GB of data What do you think rclone should do? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2539 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABfZI23xCWaerYtR94FlOMcS5xkunBgcks5ugIJ-gaJpZM4Wb0hd> .

ncw · 2018-10-01T11:14:56Z

Hmm, this is harder than it looks! The obvious thing to do is re-use the internal syncing routines, however they don't take a directory path so they need to be re-worked first which is a much bigger job!

dragetd · 2018-12-27T09:13:18Z

There are applications which primarily save data but have certain config files with a move-operation on them. Those files are small and only a few - but the software fails without being able to mv them.

I tried running those on b2 and it fails. It would be awesome if rclone had a workaround for this, even knowing that it can be very expensive sometimes.

ncw · 2018-12-29T15:22:36Z

@dragetd wrote:

There are applications which primarily save data but have certain config files with a move-operation on them. Those files are small and only a few - but the software fails without being able to mv them.

I tried running those on b2 and it fails. It would be awesome if rclone had a workaround for this, even knowing that it can be very expensive sometimes.

I see your point. It would be very easy to implement, just turn this into a log of some kind

https://github.com/ncw/rclone/blob/3220acc72900aa4d56b36871bae94e9145b89cac/vfs/file.go#L112-L116

What would you think if rclone logged an ERROR for every rename something like

"Fs %q can't rename files (no server side Move or Copy) do downloading and re-uploading the file"

Instead of failing? Perhaps that would be the best approach overall and not need a flag to turn it off. Especially since b2 is the only backend that doens't do server side move or copy (apart from http which is read only): https://rclone.org/overview/#optional-features

dragetd · 2019-01-06T15:57:13Z

I contacted b2 support a few days ago. They said they are thinking about implementing a move API during this year.

But I have given up on b2 anyways and just moved to wasabi.com - they offer similar prices and do support moving files around. :)

-edit- little correction. DirMove is also not supported, also not via the caching provider. It again does not work with borg, which is a pity. :(

This does the equivalent of sync.Move but is specialised for moving files in one backend.

…2539 Previously to this change, backends without the optional interface DirMove could not rename directories. This change uses the new operations.DirMove call to implement renaming directories which will fall back to Move/Copy as necessary.

This does the equivalent of sync.Move but is specialised for moving files in one backend.

…2539 Previously to this change, backends without the optional interface DirMove could not rename directories. This change uses the new operations.DirMove call to implement renaming directories which will fall back to Move/Copy as necessary.

ncw · 2019-01-27T21:30:06Z

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.46

This does the equivalent of sync.Move but is specialised for moving files in one backend.

…clone#2539 Previously to this change, backends without the optional interface DirMove could not rename directories. This change uses the new operations.DirMove call to implement renaming directories which will fall back to Move/Copy as necessary.

This does the equivalent of sync.Move but is specialised for moving files in one backend.

…clone#2539 Previously to this change, backends without the optional interface DirMove could not rename directories. This change uses the new operations.DirMove call to implement renaming directories which will fall back to Move/Copy as necessary.

tcf909 changed the title ~~RCLONE MOUNT - Move & DirMove on unsupported backends (mimic with copy/remove)~~ RCLONE MOUNT backends without support for DirMove and Move do not fall back to copy or upload / download Sep 5, 2018

tcf909 changed the title ~~RCLONE MOUNT backends without support for DirMove and Move do not fall back to copy or upload / download~~ rclone mount - backends without support for DirMove and Move do not fall back to copy or upload / download Sep 5, 2018

ncw added enhancement VFS / mount labels Oct 1, 2018

ncw added this to the v1.44 milestone Oct 1, 2018

ncw modified the milestones: v1.44, v1.45 Oct 19, 2018

ncw modified the milestones: v1.45, v1.46 Nov 24, 2018

ncw added a commit that referenced this issue Jan 15, 2019

operations: Implement DirMove for moving a directory #2539

4801c0b

This does the equivalent of sync.Move but is specialised for moving files in one backend.

ncw added a commit that referenced this issue Jan 27, 2019

operations: Implement DirMove for moving a directory #2539

bbd03f4

This does the equivalent of sync.Move but is specialised for moving files in one backend.

ncw closed this as completed Jan 27, 2019

gcornut pushed a commit to gcornut/rclone that referenced this issue Mar 15, 2019

operations: Implement DirMove for moving a directory rclone#2539

614d045

This does the equivalent of sync.Move but is specialised for moving files in one backend.

gcornut pushed a commit to gcornut/rclone that referenced this issue Mar 15, 2019

operations: Implement DirMove for moving a directory rclone#2539

5bc6bf0

This does the equivalent of sync.Move but is specialised for moving files in one backend.

dquintela mentioned this issue May 21, 2019

Backblaze: Implement CopyFile #3215

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rclone mount - backends without support for DirMove and Move do not fall back to copy or upload / download #2539

rclone mount - backends without support for DirMove and Move do not fall back to copy or upload / download #2539

tcf909 commented Sep 5, 2018 •

edited

tcf909 commented Sep 5, 2018 •

edited

tcf909 commented Sep 5, 2018

brechmos commented Sep 8, 2018 •

edited

ncw commented Sep 11, 2018

brechmos commented Sep 11, 2018

brechmos commented Sep 11, 2018 •

edited

ncw commented Sep 11, 2018

tcf909 commented Sep 11, 2018

brechmos commented Sep 11, 2018

ncw commented Sep 29, 2018

tcf909 commented Sep 29, 2018 via email

ncw commented Sep 30, 2018

tcf909 commented Sep 30, 2018 via email

ncw commented Oct 1, 2018

dragetd commented Dec 27, 2018

ncw commented Dec 29, 2018

dragetd commented Jan 6, 2019 •

edited

ncw commented Jan 27, 2019

rclone mount - backends without support for DirMove and Move do not fall back to copy or upload / download #2539

rclone mount - backends without support for DirMove and Move do not fall back to copy or upload / download #2539

Comments

tcf909 commented Sep 5, 2018 • edited

tcf909 commented Sep 5, 2018 • edited

tcf909 commented Sep 5, 2018

brechmos commented Sep 8, 2018 • edited

ncw commented Sep 11, 2018

brechmos commented Sep 11, 2018

brechmos commented Sep 11, 2018 • edited

ncw commented Sep 11, 2018

tcf909 commented Sep 11, 2018

brechmos commented Sep 11, 2018

ncw commented Sep 29, 2018

tcf909 commented Sep 29, 2018 via email

ncw commented Sep 30, 2018

tcf909 commented Sep 30, 2018 via email

ncw commented Oct 1, 2018

dragetd commented Dec 27, 2018

ncw commented Dec 29, 2018

dragetd commented Jan 6, 2019 • edited

ncw commented Jan 27, 2019

tcf909 commented Sep 5, 2018 •

edited

tcf909 commented Sep 5, 2018 •

edited

brechmos commented Sep 8, 2018 •

edited

brechmos commented Sep 11, 2018 •

edited

dragetd commented Jan 6, 2019 •

edited