Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
[architecture] Performance of `restic snapshots` with high-latency remote #523
Hey, thanks for raising this issue. As you already discovered, this patch will only divide the time needed to list all snapshots by four, so it's not a long-term fix. If you want to create a PR to mitigate the issue for some time, please go ahead. In order to get that PR merged, please have a look at the worker pool implementation in
For a long-term fix, I plan to make a change to the repository layout and allow packed snapshots. We already have the infrastructure for packing files (and it works really well), so when the prune/optimize command (whatever it will be called) runs, it will also pack snapshots. Since the data stored for each snapshot is really tiny it allows us to pack A LOT of snapshots into one packfile.
What do you think?
It's an interesting idea. I like the reuse of the same pack system.
But it's only efficient as long as all the snapshots end up in the same pack, and, the whole pack is downloaded once and cached. If prune/optimize runs multiple times, then the snapshots will end up scattered throughout many packs. In the worst case (optimize after each backup) it's no more efficient than the current system.
I think optimize after each backup is a normal use case (e.g. backup ; delete snapshots older than 90 days ; optimize ) so this worst case would be hit often.
Maybe better to use the "supercedes" concept from the index system instead?