folders skipped from syncing

[stefano70]

Having a source including a folder where rclone has no permissions, then the directory listing stops and the sync process is not completed not only for the folder with no permissions, but even for other files and folders elsewhere around.

https://forum.rclone.org/t/directory-skipped-from-sync/797

I vote for including a flag to force rclone to skip folders without permission and go on syncing the other object. Of course if a folder in the source is not accesible for some reason, then rclone sync must not DELETE the target folder...

[ncw]

Doing it with a flag would probably be a good idea.

I propose using the flag --skip-unreadable which would mean that a file or directory that rclone can't read would be skipped. This would be implemented by the local fs, but could in theory be implemented by others such as sftp.

In general you don't want it though as if you were syncing a directory, then it became unreadable for some reason, it would get deleted on the remote.

[ishuah]

I can't reproduce this issue with the latest beta (https://beta.rclone.org/v1.38-173-g2497ca51/).
sync reports an error error reading source directory when it gets to a directory it doesn't have permission to but it doesn't fail, it just skips the directory and moves to the next. Same with files.

Looks like @ncw unintentionally fixed this issue back in September :)
march.go L120, using go routines to traverse directories and each directory is considered a job. Any error reading the directory is logged and the directory is skipped.

Sync happens.

[ncw]

Thats for looking at this @ishuah . I concur that the issue is now fixed! It will skip directories it can't read and transfer as much stuff as it can. The sync will be marked as failed, and will be retried (depending on --retries

Of course if a folder in the source is not accesible for some reason, then rclone sync must not DELETE the target folder...

This is covered already - you'll get a log which looks like this

$ rclone --retries 1 -v sync src dst
2017/11/29 09:45:37 INFO  : Local file system at /tmp/dst: Modify window is 1ns
2017/11/29 09:45:37 ERROR : bad: error reading source directory: failed to open directory "bad": open /tmp/src/bad: permission denied
2017/11/29 09:45:37 INFO  : Local file system at /tmp/dst: Waiting for checks to finish
2017/11/29 09:45:37 INFO  : Local file system at /tmp/dst: Waiting for transfers to finish
2017/11/29 09:45:37 ERROR : Local file system at /tmp/dst: not deleting files as there were IO errors
2017/11/29 09:45:37 ERROR : Local file system at /tmp/dst: not deleting directories as there were IO errors
2017/11/29 09:45:37 ERROR : Attempt 1/1 failed with 1 errors and: not deleting files as there were IO errors
2017/11/29 09:45:37 Failed to sync: not deleting files as there were IO errors