Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce memory usage for large directories/files #3773

Merged
merged 7 commits into from
Jul 23, 2022
Merged

Reduce memory usage for large directories/files #3773

merged 7 commits into from
Jul 23, 2022

Commits on Jul 23, 2022

  1. archiver: remove dead attribute from FutureNode

    @MichaelEischer
    MichaelEischer committed Jul 23, 2022
    Configuration menu
    Copy the full SHA
    79321a1 View commit details
    Browse the repository at this point in the history

  2. archiver: cleanup Saver interface

    @MichaelEischer
    MichaelEischer committed Jul 23, 2022
    Configuration menu
    Copy the full SHA
    dcb00fd View commit details
    Browse the repository at this point in the history

  3. archiver: remove unused fileInfo from progress callback

    @MichaelEischer
    MichaelEischer committed Jul 23, 2022
    Configuration menu
    Copy the full SHA
    32f4997 View commit details
    Browse the repository at this point in the history

  4. archiver: unify FutureTree/File into futureNode 

    There is no real difference between the FutureTree and FutureFile
structs. However, differentiating both increases the size of the
FutureNode struct.

The FutureNode struct is now only 16 bytes large on 64bit platforms.
That way is has a very low overhead if the corresponding file/directory
was not processed yet.

There is a special case for nodes that were reused from the parent
snapshot, as a go channel seems to have 96 bytes overhead which would
result in a memory usage regression.
    @MichaelEischer
    MichaelEischer committed Jul 23, 2022
    Configuration menu
    Copy the full SHA
    c206a10 View commit details
    Browse the repository at this point in the history

  5. archiver: Incrementally serialize tree nodes 

    That way it is not necessary to keep both the Nodes forming a Tree and
the serialized JSON version in memory.
    @MichaelEischer
    MichaelEischer committed Jul 23, 2022
    Configuration menu
    Copy the full SHA
    b817681 View commit details
    Browse the repository at this point in the history

  6. archiver: reduce memory usage for large files 

    FutureBlob now uses a Take() method as a more memory-efficient way to
retrieve the futures result. In addition, futures are now collected
while saving the file. As only a limited number of blobs can be queued
for uploading, for a large file nearly all FutureBlobs already have
their result ready, such that the FutureBlob object just consumes
memory.
    @MichaelEischer
    MichaelEischer committed Jul 23, 2022
    Configuration menu
    Copy the full SHA
    4a10ebe View commit details
    Browse the repository at this point in the history

  7. Add changelog for the optimized tree serialization

    @MichaelEischer
    MichaelEischer committed Jul 23, 2022
    Configuration menu
    Copy the full SHA
    2ba1416 View commit details
    Browse the repository at this point in the history