Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry streaming S3 object downloads using range #2326

Closed
alextwoods opened this issue Jun 10, 2020 · 0 comments · Fixed by #2343
Closed

Retry streaming S3 object downloads using range #2326

alextwoods opened this issue Jun 10, 2020 · 0 comments · Fixed by #2343
Labels
feature-request A feature should be added or improved. help wanted We are asking the community to submit a PR to resolve this issue.

Comments

@alextwoods
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Copying from #1617 :

When downloading S3 objects with a block, aws-sdk-s3 internally creates a BlockIO to be the writable response target. Normally aws-sdk-s3 retries any failed downloads in case of network errors, but not for streaming downloads. This is because BlockIO is not truncatable, unlike a response target like StringIO or Tempfile, and aws-sdk-s3 needs it to be because it retries the whole download from the start.

Describe the solution you'd like
s3 get_object supports the range parameter. On retry of streaming bodies for the get_object request, we can use the number of received so far to add a range parameter, thus avoiding re-downloading data that has already been sent.

@alextwoods alextwoods added feature-request A feature should be added or improved. help wanted We are asking the community to submit a PR to resolve this issue. labels Jun 10, 2020
@alextwoods alextwoods linked a pull request Jun 23, 2020 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request A feature should be added or improved. help wanted We are asking the community to submit a PR to resolve this issue.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant