Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory usage when backing up directories containing many empty files #2446

Open
pabs3 opened this issue Oct 19, 2019 · 0 comments

Comments

@pabs3
Copy link

@pabs3 pabs3 commented Oct 19, 2019

Output of restic version

restic 0.9.5 compiled with go1.12.9 on linux/amd64

How did you run restic exactly?

See the attached test script, which I ran like this:

$ ./test 1000000

The script creates a directory containing a million empty files and then backs it up using restic. On an ext4 filesystem the directory is 22MB in size.

What backend/server/service did you use to store the repository?

Filesystem

Expected behaviour

Lower memory usage.

Actual behaviour

Higher memory usage, see the attached heap profiles with GOGC unset and
GOGC=1 taken towards the end of the restic backup process. The peak RAM usage with GOGC unset is 2.8 GB.

With real-world workloads (such as large Maildirs containing more files), the RAM usage can exceed the amount of RAM on the system, leading to swap death.

Steps to reproduce the behaviour

See the attached test script.

Do you have any idea what may have caused this?

I am guessing that restic prepares directory indexes in RAM instead of preparing them on-disk. The issue could also be inefficiently stored directory indexes.

Do you have an idea how to solve the issue?

I don't know enough about restic's internals nor Golang memory profiling.

Did restic help you or made you happy in any way?

I was happy to find a good replacement for rdiff-backup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.