local: windows continuously copies files with bad characters

[ncw]

From the forum.

• Create a file in Google Drive using a non-compatible special character ( * or : for example)
• Sync the file using rclone for windows to a windows device (doesn't appear to occur in first sync)
• Attempt to sync again, rclone should then say that it is replacing invalid characters again, copying again, but then deletes it at the end of the run.

X:\go\src\github.com\ncw\rclone>rclone -q ls drive:wintest
        0 bad:windows:file

X:\go\src\github.com\ncw\rclone>dir /w wintest
 Volume in drive X is VBOX_ncw
 Volume Serial Number is 0000-FC01

 Directory of X:\go\src\github.com\ncw\rclone\wintest

bad_windows_file
               1 File(s)              0 bytes
               0 Dir(s)  18,959,835,136 bytes free

X:\go\src\github.com\ncw\rclone>rclone.exe sync drive:wintest wintest -v
2016/11/20 22:09:59 rclone: Version "v1.34-DEV" starting with parameters ["rclon
e.exe" "sync" "drive:wintest" "wintest" "-v"]
2016/11/20 22:10:00 Local file system at \\?\X:\go\src\github.com\ncw\rclone\win
test: Modify window is 1ms
2016/11/20 22:10:00 Google drive root 'wintest': Reading ""
2016/11/20 22:10:00 Google drive root 'wintest': Finished reading ""
2016/11/20 22:10:00 Local file system at \\?\X:\go\src\github.com\ncw\rclone\win
test: Waiting for checks to finish
2016/11/20 22:10:00 Local file system at \\?\X:\go\src\github.com\ncw\rclone\win
test: Waiting for transfers to finish
2016/11/20 22:10:01 Local file system at \\?\X:\go\src\github.com\ncw\rclone\win
test: Replacing invalid characters in "\\go\\src\\github.com\\ncw\\rclone\\winte
st\\bad:windows:file" to "\\\\?\\X:\\go\\src\\github.com\\ncw\\rclone\\wintest\\
bad_windows_file"
2016/11/20 22:10:01 bad:windows:file: Copied (new)
2016/11/20 22:10:01 Waiting for deletions to finish
2016/11/20 22:10:01 bad_windows_file: Deleted
2016/11/20 22:10:01
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 1
Transferred:            1
Elapsed time:        1.5s
2016/11/20 22:10:01 Go routines at exit 5

X:\go\src\github.com\ncw\rclone>

Solution

This isn't trivial to solve. The sync engine needs to know any transformed name on both the source and the destination to fix this, and the transforms need to be reversible.

Could have a set of forbidden characters for each fs that we must transform.

You'd have a minimum base set which would be the control characters to \u2400 + the character, and / to oblique solidus (or whatever it is called!).

local/Windows would add the characters windows normally transforms in (so * to oblique *)

OneDrive would add those characters too.

The sync engine could somehow use the union of the source and dest sets to work out what needs transforming to what.

Needs more thought!

Shoud probably write a program to determine the acceptable chars for each remote (and length of file names). Make it an rclone command?

[ivandeex]

Latest rclone encodes filenames. This issue needs a retest.

[ncw]

Yes this is definitely fixed now by filename encoding - I shall close the issue :-)