Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize prune memory usage #3899

Merged
merged 5 commits into from
Oct 22, 2022
Merged

Optimize prune memory usage #3899

merged 5 commits into from
Oct 22, 2022

Commits on Oct 22, 2022

  1. restic: Add CountedBlobSet type 

    This allows maintaining a usage counter for each blob.
    @MichaelEischer
    MichaelEischer committed Oct 22, 2022
    Configuration menu
    Copy the full SHA
    b21241e View commit details
    Browse the repository at this point in the history

  2. prune: Use a single CountedBlobSet to track blobs 

    The set covers necessary, existing and duplicate blobs. This removes the
duplicate sets used to track whether all necessary blobs also exist.
This reduces the memory usage of prune by about 20-30%.
    @MichaelEischer
    MichaelEischer committed Oct 22, 2022
    Configuration menu
    Copy the full SHA
    c4fc5c9 View commit details
    Browse the repository at this point in the history

  3. prune: Shrink keepBlobs set if possible 

    As long as only a small fraction of the data in a repository is
rewritten, the keepBlobs set will be rather small after cleaning it up.
As golang maps do not shrink their memory usage, just copy the contents
over to a new map. However, only copy the map if the cleanup removed at
least half the entries.
    @MichaelEischer
    MichaelEischer committed Oct 22, 2022
    Configuration menu
    Copy the full SHA
    68c9cb9 View commit details
    Browse the repository at this point in the history

  4. add prune memory optimization changelog

    @MichaelEischer
    MichaelEischer committed Oct 22, 2022
    Configuration menu
    Copy the full SHA
    1e2794f View commit details
    Browse the repository at this point in the history

  5. prune: allow gc of set of repacked blobs before index rebuild

    @MichaelEischer
    MichaelEischer committed Oct 22, 2022
    Configuration menu
    Copy the full SHA
    d966c52 View commit details
    Browse the repository at this point in the history