Implement cleanup for other remotes

[ncw]

Implement cleanup for remotes

• amazonclouddrive - not possible
• b2 - done in b2: clean previous file versions #462
• box - not possible easily (see below)
• drive
• dropbox - not possible
• googlecloudstorage - not possible
• hubic - not possible
• local - not possible
• onedrive - not possible
• s3 - not possible
• swift - not possible
• yandex

[cederberg]

Trying v1.34 of rclone, I can't seem to get rclone cleanup to work properly against B2. The files remain listed (hidden) in the Backblaze file browser tool (web site).

Strangely, the hidden files do not show up when running rclone cleanup -v .... A guess might be that this is the root cause. Also tried with the --b2-versions flag, but to no avail.

[ncw]

@cederberg rclone cleanup should work properly with b2 now. Can you open a new issue with a log file and some instructions on how to reproduce please? I thought this might be related to #604 but I fixed that apparently!

[cederberg]

Hmm. Tried to reproduce, but now it works. Weird. The B2 web UI is sometimes slow, but shouldn't take several minutes to update... Hmm...

Ignore this for now. I'll file a proper bug if I see this again (and can repro the issue). Thanks for the quick support!

[ncw]

@cederberg might be caching I suppose... File a bug if you can reproduce it! Thanks, Nick

[eharris]

I haven't had a chance to dig into the code yet, but I'm wondering if cleanup is smart enough to only "cleanup" the files that rclone deleted, or if it does a whole-storage empty/purge? If the latter, I'm not sure I'd be comfortable with that without a pretty clear confirmation or --yes-really kind of option.

[ncw]

@eharris cleanup will do its stuff on all files, regardless of whether rclone deleted them. It should only remove already deleted files.

If the latter, I'm not sure I'd be comfortable with that without a pretty clear confirmation or --yes-really kind of option.

Lots of rclone commands can cause data loss if used incorrectly. The --dry-run option is for those cases!

[timwhite]

Moving comments from #1489 to here:
It would be great to support Google Drive "Cleanup", or at least the ability to empty the Trash. Both V2 and V3 of the API support it https://developers.google.com/drive/v3/reference/files/emptyTrash

[juanluisbaptiste]

Any ETA on google drive ?

[ishuah]

@ncw I've been looking into cleanup for amazonclouddrive, box, dropbox and onedrive.

amazonclouddrive: There doesn't seem to be any documentation on permanently deleting trashed items with the Amazon Cloud Drive API. Docs

box: The Box API provides an endpoint for listing trashed items and permanently deleting trashed items. However, there's no endpoint for permanently deleting trashed items and the Box API does not support batch API calls. Cleanup is not feasible since it involves listing all trashed items and making a delete request for each item. Docs

dropbox: There doesn't seem to be any documentation on permanently deleting trashed items with the Dropbox API. From my understanding, it is not yet supported. Docs

onedrive: Same as dropbox and amazonclouddrive, the OneDrive API does not expose a way to permanently delete trashed items. Docs

[ncw]

@ishuah - thanks for working on this ticket and the informative info above.

I make this ticket all done now - do you agree?

[ishuah]

@ncw I agree, this ticket is done.

[ncw]

Excellent - thank you very much @ishuah

[juanluisbaptiste]

I'm getting this error with google drive:

clone cleanup gdrive-backups-3:
2018/11/20 22:27:51 Attempt 1/3 failed with 0 errors and: Google drive root '' doesn't support cleanup
2018/11/20 22:27:51 Attempt 2/3 failed with 0 errors and: Google drive root '' doesn't support cleanup
2018/11/20 22:27:51 Attempt 3/3 failed with 0 errors and: Google drive root '' doesn't support cleanup
2018/11/20 22:27:51 Failed to cleanup: Google drive root '' doesn't support cleanup

Command taken from:

https://rclone.org/drive/#emptying-trash

Any ideas ?

[juanluisbaptiste]

Ooops sorry, system where I was testing had an old release, everything is ok with latest release.

[hexxxx]

rclone cleanup remote:path not working for a google drive remote. Command returns without errors, but running rclone about remote: still shows terabytes in the trash. Any thoughts?

[ncw]

rclone cleanup remote:path not working for a google drive remote. Command returns without errors, but running rclone about remote: still shows terabytes in the trash. Any thoughts?

I think it can take some time for the cleanup to take effect.

[hexxxx]

rclone cleanup remote:path not working for a google drive remote. Command returns without errors, but running rclone about remote: still shows terabytes in the trash. Any thoughts?

I think it can take some time for the cleanup to take effect.

Well, you may be right. It's gone down about 50GB in the past 6 hours. Maybe it'll just take a while to empty. Thanks!