Add --force-cleanup option to cleanup command

[kennyparsons]

Original forum discussion: https://forum.rclone.org/t/empty-google-drive-trash/14331/5

What is your current rclone version (output from rclone version)?

rclone v1.51.0
- os/arch: linux/amd64
- go version: go1.13.7

What problem are you are trying to solve?

The cleanup command has some nebulous results, as seen in several of the forum posts. An alternative, more manual command I had great success with was:
rclone delete gdrive: --drive-trashed-only --drive-use-trash=false --verbose=2 --fast-list

This command enumerated and manually deleted several terabytes of junk data sitting the trash, whereas the cleanup option was slow and results have varied for many.

How do you think rclone should be changed to solve that?

I would like to see an additional flag, such as --force-cleanup, available to the cleanup command. This flag would essentially be a shortcut to flip the cleanup command to a delete command with --drive-trashed-only and --drive-use-trash=false as default attached commands. The --forced-cleanup flag ideally would also accept additional, existing flags, such as --fast-list, --verbose=x, etc.

[biship]

Agree. cleanup doesn't seem to clean up everything, or doesn't work as intended.

What's interesting is that I can run empty trash from the GDrive web site and it shows as empty.

This command finds and deletes files that are 3+ months old:
rclone delete gdrive: --drive-trashed-only --drive-use-trash=false --verbose=2 --fast-list

[ncw]

The cleanup command has some nebulous results, as seen in several of the forum posts.

The cleanup call, calls Googles API to empty the trash which seems a bit nebulous. It sometimes takes days to work.

An alternative, more manual command I had great success with was: rclone delete gdrive: --drive-trashed-only --drive-use-trash=false --verbose=2 --fast-list

This command enumerated and manually deleted several terabytes of junk data sitting the trash, whereas the cleanup option was slow and results have varied for many.

Well, we implemented clean up for Google Team drives with effectively that rclone delete command.

I'm pretty sure that running this code for both team drives and non team drives would do the right thing

rclone/backend/drive/drive.go

Lines 2476 to 2482 in d252816

	directoryID, err := f.dirCache.FindDir(ctx, "", false)
	if err != nil {
	return err
	}
	directoryID = actualID(directoryID)
	_, err = f.cleanupTeamDrive(ctx, "", directoryID)
	return err

Fancy giving that a go @biship or @kennyparsons ?

[np-tokumei]

Hello! Thanks for proposing this command. It did show my LOTS of files in my trash that could be deleted when I have --dry-run. However, when running the actual command (i.e., removing --dry-run), I got these errors:

2023/04/24 22:46:35 DEBUG : Waiting for deletions to finish
2023/04/24 22:47:05 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:47:05 DEBUG : pacer: Rate limited, increasing sleep to 1.304474323s
2023/04/24 22:47:05 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:47:05 DEBUG : pacer: Rate limited, increasing sleep to 2.235448938s
2023/04/24 22:47:05 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:47:05 DEBUG : pacer: Rate limited, increasing sleep to 4.445288018s
2023/04/24 22:47:05 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:47:05 DEBUG : pacer: Rate limited, increasing sleep to 8.481838849s
2023/04/24 22:47:34 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:47:34 DEBUG : pacer: Rate limited, increasing sleep to 16.72302376s
2023/04/24 22:47:36 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:47:36 DEBUG : pacer: Rate limited, increasing sleep to 16.214804273s
2023/04/24 22:47:44 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:47:44 DEBUG : pacer: Rate limited, increasing sleep to 16.842560309s
2023/04/24 22:47:53 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:47:53 DEBUG : pacer: Rate limited, increasing sleep to 16.019309325s
2023/04/24 22:48:03 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:48:03 DEBUG : pacer: Rate limited, increasing sleep to 16.391229821s
2023/04/24 22:48:20 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:48:20 DEBUG : pacer: Rate limited, increasing sleep to 16.878197542s
2023/04/24 22:48:37 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:48:37 DEBUG : pacer: Rate limited, increasing sleep to 16.29832887s
2023/04/24 22:48:53 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:48:53 DEBUG : pacer: Rate limited, increasing sleep to 16.086835185s
2023/04/24 22:49:10 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:49:10 DEBUG : pacer: Rate limited, increasing sleep to 16.521523722s
2023/04/24 22:49:27 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:49:27 DEBUG : pacer: Rate limited, increasing sleep to 16.329183349s
2023/04/24 22:49:43 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:49:43 DEBUG : pacer: Rate limited, increasing sleep to 16.770972947s
2023/04/24 22:49:59 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:49:59 DEBUG : pacer: Rate limited, increasing sleep to 16.559269939s
2023/04/24 22:50:15 DEBUG : pacer: low level retry 5/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:50:15 DEBUG : pacer: Rate limited, increasing sleep to 16.280811851s
2023/04/24 22:50:32 DEBUG : pacer: low level retry 5/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:50:32 DEBUG : pacer: Rate limited, increasing sleep to 16.540360888s
2023/04/24 22:50:49 DEBUG : pacer: low level retry 5/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:50:49 DEBUG : pacer: Rate limited, increasing sleep to 16.303100919s
2023/04/24 22:51:05 DEBUG : pacer: low level retry 5/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:51:05 DEBUG : pacer: Rate limited, increasing sleep to 16.227588179s
2023/04/24 22:51:22 DEBUG : pacer: low level retry 6/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:51:22 DEBUG : pacer: Rate limited, increasing sleep to 16.076963957s
2023/04/24 22:51:38 DEBUG : pacer: low level retry 6/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:51:38 DEBUG : pacer: Rate limited, increasing sleep to 16.324983497s
2023/04/24 22:51:54 DEBUG : pacer: low level retry 6/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:51:54 DEBUG : pacer: Rate limited, increasing sleep to 16.143021953s
2023/04/24 22:52:10 DEBUG : pacer: low level retry 6/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:52:10 DEBUG : pacer: Rate limited, increasing sleep to 16.058486466s
2023/04/24 22:52:27 DEBUG : pacer: low level retry 7/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:52:27 DEBUG : pacer: Rate limited, increasing sleep to 16.510395638s
2023/04/24 22:52:43 DEBUG : pacer: low level retry 7/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:52:43 DEBUG : pacer: Rate limited, increasing sleep to 16.516254239s
2023/04/24 22:52:59 DEBUG : pacer: low level retry 7/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:52:59 DEBUG : pacer: Rate limited, increasing sleep to 16.070558931s
2023/04/24 22:53:15 DEBUG : pacer: low level retry 7/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:53:15 DEBUG : pacer: Rate limited, increasing sleep to 16.92407376s
2023/04/24 22:53:32 DEBUG : pacer: low level retry 8/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:53:32 DEBUG : pacer: Rate limited, increasing sleep to 16.21391181s
2023/04/24 22:53:48 DEBUG : pacer: low level retry 8/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:53:48 DEBUG : pacer: Rate limited, increasing sleep to 16.617681404s
2023/04/24 22:54:05 DEBUG : pacer: low level retry 8/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:54:05 DEBUG : pacer: Rate limited, increasing sleep to 16.090486973s
2023/04/24 22:54:21 DEBUG : pacer: low level retry 8/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:54:21 DEBUG : pacer: Rate limited, increasing sleep to 16.828332426s
2023/04/24 22:54:38 DEBUG : pacer: low level retry 9/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:54:38 DEBUG : pacer: Rate limited, increasing sleep to 16.23622363s
2023/04/24 22:54:54 DEBUG : pacer: low level retry 9/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:54:54 DEBUG : pacer: Rate limited, increasing sleep to 16.542527962s
2023/04/24 22:55:11 DEBUG : pacer: low level retry 9/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:55:11 DEBUG : pacer: Rate limited, increasing sleep to 16.339000606s
2023/04/24 22:55:27 DEBUG : pacer: low level retry 9/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:55:27 DEBUG : pacer: Rate limited, increasing sleep to 16.39761947s
2023/04/24 22:55:43 DEBUG : pacer: low level retry 10/10 (error googleapi: Error 500: Internal Error, internalError)
2023/04/24 22:55:43 DEBUG : pacer: Rate limited, increasing sleep to 16.619374543s
2023/04/24 22:55:43 ERROR : file_name: Couldn't delete: googleapi: Error 500: Internal Error, internalError
2023/04/24 22:56:00 DEBUG : pacer: low level retry 10/10 (error googleapi: Error 500: Internal Error, internalError)

Checking my console, this shows all "drive.files.delete" method were responded with 100% errors.

any suggestion is greatly appreciated.