Does Rustic use the same index structure as Restic? Has it been changed to improve memory management?

[preranaprabhu02]

I feel restic takes a long time to load and re-load the indexes and this process is tedious.

[aawsome]

Do you get the same behavior with rustic? Is there something we should change in rustic?

About your question:

• The repository index file format is almost identical to restic, but rustic added (optional) fields to support some features restic doesn't have, like the lock free pruning or prune --keep-pack.
• The in-memory index representation is quite optimized in rustic, mainly to get a lower memory profile. The memory management functionality of Rust also plays a big role here.

[preranaprabhu02]

Thanks for the clarification!! It did give a little insight. Also stumbled upon Rustic, will be happy to try it out soon!!

[aawsome]

A bit more insight:

• rustic uses a sorted array to store the index which it reads from the repository (and basically a hashmap for newly created index entries) whereas restic uses a hashmap for all index entries. As the stored index is static during a rustic run, the sorting needs to be done only at the beginning, is parallelized and therefore very fast. Storing in an array reduces the memory consumtion quite a lot. In theory using a sorted array, the access is slower for large indexes but in practice it works really well without being a bottleneck.
• rustic saves only the parts of the index in memory which is needed for the command to run. For instance, rustic backup only needs to know if a data blob already exists in the repository but not the actual pack file and the position within. This again reduces the memory footprint a lot.
• The lack of the garbage collector reduces the memory footprint of large dynamic data structures as unneeded data is removed immediately from memory.

I turn this issue into a discussion as it is better fitted there.