Use Content-Range in multipart download for totalBytes #5101
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… the total bytes of the TransferPRogress.
|
zoewangg
reviewed
Apr 11, 2024
|
|
||
| /** | ||
| * Parse the Content-Range to extract the total number of byte from the content. Only supports the {@code bytes} unit, any | ||
| * other unit will result in an empty OptionalLong. If the total length in unknown, which is represented by a {@code *} symbol |
There was a problem hiding this comment.
Just curious, in what scenarios, total length could be unknown (*)?
There was a problem hiding this comment.
It is part of the reference spec. I'm not sure if S3 uses it.
zoewangg
reviewed
Apr 11, 2024
.../main/java/software/amazon/awssdk/transfer/s3/internal/progress/TransferProgressUpdater.java
Outdated
Show resolved
Hide resolved
… the total bytes of the TransferPRogress.
zoewangg
approved these changes
Apr 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change
TransferProgressUpdaterso it uses Content-Range header to determine the maximum number of bytes it tracks as part of the transfer progress listener.Motivation and Context
For downloads,
TransferProgressUpdaterwas expecting the totalBytes of the whole s3 object to be the value of theContent-Lengthheader, when in fact theContent-Lengthheader only contains the size of the current part. This cause the value of transferred bytes to exceed to the number total byte expected as soon as a 2nd part was dowloaded.Modifications
Add a
ContentRangeParserclass to extract the total object size from theContent-Rangeheader, following the MDN refgerence. Only supportbytesunit.Testing