Uncap delete worker concurrency for dramatic prune speed-ups [0.12.1]

[whereisaaron]

Output of restic version

restic 0.12.1 compiled with go1.16.6 on linux/amd64

What should restic do differently? Which functionality do you think we should add?

Enable users to configure delete parallelism, especially for 'prune' operations, where large numbers of items need to be deleted. Possibly by replacing the current hard-coded limit of 8 with with a configurable -o option.

There's been many improvements to concurrency in restic functions however delete parallelism appears to be capped at no more than 8 concurrent deletes. This bites hard for pruning with high network latency. My testing with B2 with ~180ms latency and maxed out settings (--max-unused unlimited -o b2.connections=8) shows that deletes are capped at ~10 deletes/second with ~180ms latency. In a large prune you're deleting close to 100,000 objects and that adds ~3 hours to the prune process.

If deletes could be configured higher that 8, e.g. to 32 or 64 workers, that 3 hours could be reduced to <30 minutes. However, users can't configure this right now. According to the community forum this is because of this hard-coded setting that caps delete workers at 8 with no configuration option to change that. So for B2 storage, even if you configure -o b2.connections=32 nothing changes, because the delete workers appear to still be capped at 8.

restic/cmd/restic/delete.go

Line 21 in bf9c877

const numDeleteWorkers = 8

Forum discussion:
https://forum.restic.net/t/o-b2-connections-n-seems-to-have-no-impact/1260/28?u=whereisaaron

What are you trying to do? What problem would this solve?

Enable restic to efficiently prune repositories in high latency networks/filesystems. And general speed up restic operations for high latency networks/filesystems.

Did restic help you today? Did it make you happy in any way?

It's amazing how fast the restores are now, they've improved so much in the last year and work well even for high latency networks/filesystems 🎉

[MichaelEischer]

Please have a look at #3611 .