Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

restorer: allow writing target file blobs out of order #2195

Open
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
3 participants
@ifedorenko
Copy link
Contributor

ifedorenko commented Mar 3, 2019

What is the purpose of this change? What does it change?

Much simpler implementation that guarantees each required pack
is downloaded only once (and hence does not need to manage
pack cache). Also improves large file restore performance.

Was the change discussed in an issue or in the forum before?

See #2074
https://forum.restic.net/t/restore-using-rclone-gdrive-backend-is-slow/1112/8
https://forum.restic.net/t/degraded-restore-performance-s3-backend/1400

Checklist

  • I have read the Contribution Guidelines
  • I have added tests for all changes in this PR
  • I have added documentation for the changes (in the manual)
  • There's a new file in changelog/unreleased/ that describes the changes for our users (template here)
  • I have run gofmt on the code in all commits
  • All commit messages are formatted in the same style as the other commits in the repo
  • I'm done, this Pull Request is ready for review

@ifedorenko ifedorenko referenced this pull request Mar 3, 2019

Closed

Out order restore #2101

5 of 7 tasks complete
@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Mar 3, 2019

Codecov Report

Merging #2195 into master will decrease coverage by 5.18%.
The diff coverage is 67.7%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2195      +/-   ##
==========================================
- Coverage   50.87%   45.68%   -5.19%     
==========================================
  Files         172      172              
  Lines       14230    14019     -211     
==========================================
- Hits         7239     6405     -834     
- Misses       5935     6616     +681     
+ Partials     1056      998      -58
Impacted Files Coverage Δ
internal/restorer/restorer.go 42.85% <100%> (+0.57%) ⬆️
internal/restorer/filerestorer.go 66.02% <63.84%> (-10.04%) ⬇️
internal/restorer/fileswriter.go 72.09% <82.75%> (-11.25%) ⬇️
internal/backend/b2/b2.go 0% <0%> (-80.69%) ⬇️
internal/backend/swift/swift.go 0% <0%> (-78.83%) ⬇️
internal/backend/gs/gs.go 0% <0%> (-74%) ⬇️
internal/backend/azure/azure.go 0% <0%> (-69.46%) ⬇️
internal/backend/swift/config.go 34.69% <0%> (-57.15%) ⬇️
internal/checker/checker.go 64.54% <0%> (-3.87%) ⬇️
internal/restic/progress_unix_with_siginfo.go 62.5% <0%> (ø)
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 870e758...9491b27. Read the comment docs.

@ifedorenko

This comment has been minimized.

Copy link
Contributor Author

ifedorenko commented Mar 3, 2019

@fd0 @mholt can you review this PR when you get a chance? It is the code discussed in #2074, and should be ready or almost ready to be merged to master.

Compared to master, this PR does not guarantee target files are written sequentially, start to finish, which allowed much simpler implementation and faster multithreaded restore of very large files.

@mholt

This comment has been minimized.

Copy link
Contributor

mholt commented Mar 3, 2019

Sure -- I just barely finished my masters thesis so I'm in major catch-up mode, but at a glance I really like the ~1K LoC reduction. Don't wait up on me, but I will try to give it a high-level once over soon.

restorer: allow writing target file blobs out of order
Much simpler implementation that guarantees each required pack
is downloaded only once (and hence does not need to manage
pack cache). Also improves large file restore performance.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>

@ifedorenko ifedorenko force-pushed the ifedorenko:out-of-order-restore-no-progress branch from f4c7335 to 9491b27 Mar 17, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.