-
-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chunker: implement Resumer interface #5185
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
9a7e3db
to
20a47e0
Compare
This comment has been minimized.
This comment has been minimized.
20a47e0
to
54a24c8
Compare
@mcalman The pre-state of commits was saved at ivandeex/mcalman-chunker-implement-resumer. Please add relevant unit tests! UPDATE 2021-07-23 BTW, #4547 added a test where source does not change during interrupt. It would be good to also have a test where users attempts to resume from modified source, verify that resume machinery rejects it and think whether in result it feels convenient for end user. |
54a24c8
to
572719b
Compare
@ivandeex I added an adapted version of the test from #4547 as well as a new version of the test that uses a modified source. Currently, if a user has resuming enabled with --resume-larger and a modified source file is detected the resume is skipped for that file and the file is uploaded from scratch (a debug message also announces this). |
@ncw This one is ready. Please give it your |
572719b
to
968b7d6
Compare
968b7d6
to
1425b2d
Compare
rebased on fresh #4547 resolving a conflict after recent master merges |
1425b2d
to
844358c
Compare
cc @ncw |
I can't give a concrete answer in view of @ncw review delays increasing unpredictably.
I published a release on my repo fork with this goal in the mind. You could advertise your development on rclone forum to attract more volunteers. If you could demonstrate a convenient use case e.g. resuming a large download to local disk, it would bring more attention. |
@ncw 🙏 🙏 🙏 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm happy for you to rebase and merge this after we've merged #4547 :-)
844358c
to
294a132
Compare
Rebased on top of current #4547 |
294a132
to
b1ff25c
Compare
Added an interface for resuming failed uploads. Later on this interface can be implemented by any backend where resumes are possible. Fixes rclone#87 History of squashed commits: - minor revisions made - factored resume code to make Copy more readable - limited cache cleaning to once per file uploaded - finished refactoring - added support for cache cleaning by total size - improved cache cleaning - moved cache cleaing to new function - resume functions moved to seperate file - added resume larger option - added hashName and hashState to resume JSON - minor fixes to error handling - fixed consistency issue with fingerprints - removed resume stub from chunker - tweaked use of resume-larger option - added local implementation and initial test - fixed failing code quality tests - used atexit for signal handling across OSs - fixed for windows - fix resume test after rebase - fixed context convention violation and improved resume test - increased test content size - simulate larger reads in resume test - fixed code quality checks
- report errors if any - prevent double invocation - prefer saved hash type when many are supported
b1ff25c
to
a4b8504
Compare
@mcalman which in turn is waiting for @ncw to review which is spin off of which is stuck forever. I am sure we can solve it |
What is the purpose of this change?
Implement the Resumer interface for the Chunker backend. Thus, making resumable uploads possible for Chunker.
Was the change discussed in an issue or in the forum before?
Fixes #5154
Based on pull request #4547 (add Resumer interface)
Related to feature request #87 (Resume uploads)
Related to thread https://forum.rclone.org/t/intelligent-faster-chunker-file-updates-on-checksum-enabled-remotes/22313/7
Orthogonal to feature request #5041 (multi-thread uploads in chunker)
Orthogonal to discussion #4798 (multi-thread uploads for different backends)
Checklist