Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3RangeReader totalLength Improvment #3025

Merged
merged 4 commits into from Jul 19, 2019
Merged

S3RangeReader totalLength Improvment #3025

merged 4 commits into from Jul 19, 2019

Conversation

jbouffard
Copy link

@jbouffard jbouffard commented Jul 16, 2019 •
edited by pomadchin

Overview

This PR changes how the totalLength value of S3RangeReader is computed. It will now use a GetObjectResponse instead of a GetHeaderResponse. This will allow users to read these files even if they don't have permission to access its HEADER.

Checklist

  • docs/CHANGELOG.rst updated, if necessary
  • Check it on a real S3 bucket with permission limits

Closes #2889

Jacob Bouffard added 2 commits July 16, 2019 14:35
Changed how totalLength is produced in S3RangeReader
ad4c8af 
Signed-off-by: Jacob Bouffard <jbouffard@azavea.com>
Updated the CHANGELOG
05c9a84 
Signed-off-by: Jacob Bouffard <jbouffard@azavea.com>
@jbouffard
Copy link
Author

I'm not really sure there are any unit tests that can be created for this PR. It might be good to test this branch on a real world example.

S3RangeReader will now optionally read the HEADER
c5305dc 
Signed-off-by: Jacob Bouffard <jbouffard@azavea.com>
S3RangeReader will now not read the HEAD of the target object by default
9ae743f 
Signed-off-by: Jacob Bouffard <jbouffard@azavea.com>
@jbouffard
Copy link
Author

@echeipesh @pomadchin After mentioning my results yesterday, @moradology looked into it as well and found that using a responseStream worked fine for finding the contentLength. I ran the code again this morning and observed similar speeds as Nathan. I'm not sure what happened yesterday 😕 As everything was so much slower. Regardless, this PR should be good to merge now.

pomadchin
pomadchin approved these changes Jul 19, 2019
View reviewed changes
Copy link
Member

@pomadchin pomadchin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@pomadchin pomadchin merged commit ef382c9 into locationtech:master Jul 19, 2019
@echeipesh echeipesh mentioned this pull request Sep 23, 2019
Closed
@echeipesh echeipesh mentioned this pull request May 4, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@echeipesh echeipesh echeipesh left review comments

@pomadchin pomadchin pomadchin approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

S3RangeReader will always produce HEAD request
3 participants
@jbouffard @echeipesh @pomadchin