Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming s3 retry #2343

Merged
merged 13 commits into from Jun 25, 2020
Merged

Streaming s3 retry #2343

merged 13 commits into from Jun 25, 2020

Conversation

alextwoods
Copy link
Contributor

This is an attempt to implement #2326 - and support for retry S3 get_objects that are streaming w/ a Proc using range to not ge-download redundant bytes.

Issues:

  • need to set range directly on the header (This does get correctly factored into the signiture). Doesn't work to set it on the context.params[:range] because the request is already built when retried.
  • Does not work if range is passed to get_object initially. Range is a complex spec and would need to be parsed.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

  1. To make sure we include your contribution in the release notes, please make sure to add description entry for your changes in the "unreleased changes" section of the CHANGELOG.md file (at corresponding gem). For the description entry, please make sure it lives in one line and starts with Feature or Issue in the correct format.

  2. For generated code changes, please checkout below instructions first:
    https://github.com/aws/aws-sdk-ruby/blob/master/CONTRIBUTING.md

Thank you for your contribution!

@mullermp
Copy link
Contributor

Needs testing of some sort.. looks good otherwise.

@mullermp mullermp marked this pull request as ready for review June 23, 2020 19:22
@github-actions
Copy link

You have made a change to core without a corresponding change to the CHANGELOG.md. This change will not result in a new version and will not published unless an entry is added to CHANGELOG.md

Copy link
Contributor

@mullermp mullermp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Looks good.

gems/aws-sdk-s3/spec/plugins/streaming_retry_spec.rb Outdated Show resolved Hide resolved
@mullermp mullermp merged commit 740f929 into master Jun 25, 2020
@mullermp mullermp deleted the streaming_s3_retry branch June 25, 2020 16:36
@uthark
Copy link
Contributor

uthark commented Jun 25, 2020

Breaks with uninitialized constant Aws::S3::Plugins::RetryableBlockIO::Forwardable:
Opened PR to fix: #2348

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Retry streaming S3 object downloads using range
3 participants