New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

check: Improve cache handling #1696

Merged
merged 4 commits into from Apr 1, 2018

Conversation

Projects
None yet
5 participants
@fd0
Copy link
Member

fd0 commented Mar 31, 2018

For safety reasons, restic does not use a local metadata cache for the restic check command, so that data is loaded from the repository and restic can check it's in good condition. When the cache is disabled, restic will fetch each tiny blob needed for checking the integrity using a separate backend request. For non-local backends, that will take a long time, and depending on the backend (e.g. B2) may also be much more expensive.

This PR adds a few commits which will change the behavior as follows:

  • When restic check is called without any additional parameters, it will build a new cache in a temporary directory, which is removed at the end of the check. This way, we'll get readahead for metadata files (so restic will fetch the whole file when the first blob from the file is requested, this reduces the number of backend requests tremendously), but all data is freshly fetched from the storage backend.

  • When restic check is called with --with-cache, the default on-disc cache is used. This behavior hasn't changed since the cache was introduced.

  • When --no-cache is specified, restic falls back to the old behavior, and read all tiny blobs in separate requests.

Closes #1665, #1694

@fd0 fd0 referenced this pull request Mar 31, 2018

Closed

Build new cache during check #1665

@fd0 fd0 force-pushed the fix-check-cache branch from 5561377 to ffd7f18 Mar 31, 2018

@fd0 fd0 force-pushed the fix-check-cache branch from ffd7f18 to 1f5137a Apr 1, 2018

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Apr 1, 2018

Codecov Report

Merging #1696 into master will decrease coverage by 0.22%.
The diff coverage is 25.35%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1696      +/-   ##
==========================================
- Coverage   52.34%   52.11%   -0.23%     
==========================================
  Files         148      148              
  Lines       11639    11682      +43     
==========================================
- Hits         6092     6088       -4     
- Misses       4606     4651      +45     
- Partials      941      943       +2
Impacted Files Coverage Δ
internal/repository/repository.go 54.25% <17.02%> (-1.68%) ⬇️
internal/checker/checker.go 65.19% <25%> (-4.43%) ⬇️
cmd/restic/cmd_check.go 46.89% <45%> (-1.14%) ⬇️
internal/backend/rclone/backend.go 70.32% <0%> (-1.94%) ⬇️
cmd/restic/global.go 28.27% <0%> (-0.59%) ⬇️
internal/archiver/archiver.go 64.25% <0%> (+0.21%) ⬆️
internal/repository/master_index.go 47.69% <0%> (+3.07%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 22d5061...1f5137a. Read the comment docs.

@fd0 fd0 merged commit 1f5137a into master Apr 1, 2018

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

fd0 added a commit that referenced this pull request Apr 1, 2018

@fd0 fd0 deleted the fix-check-cache branch Apr 1, 2018

@askielboe

This comment has been minimized.

Copy link
Contributor

askielboe commented Apr 1, 2018

So I wanted to run a quick test against the 0.8.3 release on my B2 repo containing ~ 200 GB in 40 k files. I ran restic check with no additional arguments.

I ended up cancelling the 0.8.3 release after 26 hours, while a build including this pull request completed successfully in a bit under 15 minutes:

  • master (restic 0.8.3) - cancelled manually after 26 hours:

    • time: restic check 2128.88s user 653.90s system 2% cpu 25:58:24.15 total
    • data: sent: 824 MB (2 M packages), received: 11.69 GB (10 M packages)
  • fix-check-cache - completed successfully in 15 min:

    • time: 2553.88s user 346.09s system 325% cpu 14:51.61 total
    • data: sent: 723 KB (1,824 packages), received: 1.67 GB (1,265,453 packages)

The improvements seem quite significant. :)

@fd0

This comment has been minimized.

Copy link
Member

fd0 commented Apr 1, 2018

Uh wow, thanks for the experience report! I suspected that in some situations the improvements may be dramatic. ;)

@mathiasnagler

This comment has been minimized.

Copy link

mathiasnagler commented Apr 3, 2018

This has a dramatic effect on me as well.
Using a repository on Google Drive (over rclone), without this I cancelled the check after over 2h. With the change in place, the check finished in ~30 seconds.
Thank you very much, this is great!

@stevesbrain

This comment has been minimized.

Copy link

stevesbrain commented Apr 3, 2018

Sounds like this will affect me significantly too then, as I'm same boat as @askielboe (slightly bigger repo size, but not too far different). Excited to try it out :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment