"Duplicate file detected" problem

[yurivict]

I see this strange message:

2016/01/29 06:15:11 backup-2016-01-28_21:20:00: Duplicate file detected

Obviously, the local file system doesn't allow duplicates. But the same file is shown twice in the Google Drive through the web interface. All files only get there through the API via rclone.

[ncw]

I think this is #28

I have been unable to reproduce this problem, but if you can get a log of it happening (with the -v flag) then post it to that ticket!

The best thing to do is delete the duplicates rclone detects in the web interface.

Thanks

Nick

[ncw]

...I just release rclone 1.27 from which you can use rclone dedupe to get rid of the duplicates.

[MorganAntonsson]

I have this problem as well. I tried dedupe and it works great. I have literally thousands of duplicated files, so I wrote a little expect script to automate the work of selecting to keep the first file. I am sure it would be appreciated if you added some force flag to rclone to do that automatically. It's a great little tool you have made. Thanks. :)

[J0s3f]

I had the same problem and wrote a python/pexpect script:

#!/usr/bin/env python2.7

import pexpect

child = pexpect.spawn('rclone dedup gdrive:folder')

completed = False
while not completed:
    index = child.expect(['s/k/r>', 'Enter the number of the file to keep>', pexpect.EOF, pexpect.TIMEOUT],timeout=2600)
    if index == 0:
        child.sendline('k')
        print child.before
    elif index == 1:
        child.sendline('1')
        print child.before
        print 'duplicate removed'
    elif index == 2:
        print child.before
        print 'EOF encountered'
        completed = True
    elif index == 3:
        print child.before
        print 'TIMEOUT encountered'
        completed = True

It would be appreciated to have this as option.

[ncw]

@J0s3f I've done this already in #338 - you can find instructions and a beta release to try it on that issue.

Cool script by the way!

Thanks

Nick