Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to pause/resume s3util uploads #23

Closed
wants to merge 3 commits into from

Conversation

schmichael
Copy link

Problem:

Right now there's no way to gracefully shutdown a process mid-upload. The upload struct is private - as are a number of key pieces of state.

Proposed Solution:

Add a Pause method and a Resume function to allow for graceful shutdowns and starting back up.

Implementation details:

  • The Public API has not changed. However, I've expanded the breadth of the public API which may make compatibility harder to maintain in the future.
  • The return value of Pause, UploaderState, makes every field public to allow users to marshal it as they see fit. (Although I haven't written custom marshaling code yet, so I wonder if the part type will need to be made public to support some use cases.)
  • Resume requires a Config in addition to the UploaderState since http clients can't be automatically marshalled (not to mention callers may not want their sensitive credentials included in the uploader state).
  • I wrote a number of integration tests locally using my own S3 keys, but I didn't commit them because they're slow and require S3 keys. If you're curious, here's most of the code: https://gist.github.com/schmichael/a888d7ce5a0a9182d7c9

@schmichael
Copy link
Author

Just cleaning up old cruft of mine. 👋

@schmichael schmichael closed this Aug 24, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant