[TM DownloadFile Pause and Resume] Part 3: Implement resumeDownloadFile #3157
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zoewangg
force-pushed
the
zoewang/tm-download-resume
branch
from
e0a702c
to
4934726
Compare
zoewangg
commented
Apr 12, 2022
Comment on lines
84
to
87
| public static FileTransformerConfiguration defaultCreateOrReplaceExisting() { | ||
| return builder().fileWriteOption(FileWriteOption.CREATE_OR_REPLACE_EXISTING) | ||
| .failureBehavior(FailureBehavior.DELETE) | ||
| .failureBehavior(FailureBehavior.LEAVE) | ||
| .build(); |
There was a problem hiding this comment.
We had to change the default behavior for createOrReplaceExisting because always deleting a file upon exceptions would not work for resume (manifested as cancellation exception)
Another option is to move this method defaultCreateOrReplaceExisting along with defaultCreateOrAppend to s3-transfer-manager` and make them private APIs if we think this behavior is confusing.
WDYT?
There was a problem hiding this comment.
No I think this behavior is fine
...tom/s3-transfer-manager/src/main/java/software/amazon/awssdk/transfer/s3/ObjectTransfer.java
Outdated
Show resolved
Hide resolved
zoewangg
commented
Apr 12, 2022
Comment on lines
-85
to
-91
| public String toString() { | ||
| return ToString.builder("ResumableFileDownload") | ||
| .add("downloadFileRequest", downloadFileRequest) | ||
| .add("bytesTransferred", bytesTransferred) | ||
| .add("lastModified", lastModified) | ||
| .add("transferSizeInBytes", transferSizeInBytes) | ||
| .build(); |
There was a problem hiding this comment.
This method will be implemented in a separate PR as part of serialization/deserialization APIs
dagnir
approved these changes
Apr 13, 2022
zoewangg
force-pushed
the
zoewang/tm-download-resume
branch
from
7650a0e
to
035838c
Compare
|
Kudos, SonarCloud Quality Gate passed! |
12 tasks
zoewangg
added a commit
that referenced
this pull request
Apr 27, 2022
* [TM DownloadFile Pause and Resume] Part 1: Add configuration to enable overwriting existing files (#3125) * Expose an option to overwrite an existing file in FileAsyncResponseTransformer * Add changelog entries and make TM use CREATE_OR_REPLACE_EXISTING write option by default * Address feedback * Update and address feedback * [TM DownloadFile Pause and Resume] Part 2: Implement pause for downloadFile operation (#3094) * Part 1: Implement pause for downloadFile operation * Address feedback * Refactor the logic * Address feedback * Fix merging issue * [TM DownloadFile Pause and Resume] Part 3: Implement resumeDownloadFile (#3157) * Implement resumeDownloadFile * Move test code around * Address feedback * Fix flaky test * fix flaky integ test * add changelog entry * Troubleshooting flaky test * Add file length check when checking if file has modified or not * Address feedback
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Modifications
Implement
S3TransferManager#resumeDownloadFile.Below is the workflow for resume
Testing
Added unit tests and functional tests