[TM DownloadFile Pause and Resume] Part 2: Implement pause for downloadFile operation #3094
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
...ustom/s3-transfer-manager/src/main/java/software/amazon/awssdk/transfer/s3/FileDownload.java
Outdated
Show resolved
Hide resolved
Comment on lines
31
to
32
| public final class PersistableFileDownload implements PersistableTransfer, | ||
| ToCopyableBuilder<PersistableFileDownload.Builder, PersistableFileDownload> { |
There was a problem hiding this comment.
How can users serialize this to, e.g., a disk or database?
There was a problem hiding this comment.
I think they can use any serialization/deserialization library to do that.
There was a problem hiding this comment.
It seems like we should consider the option of vending an opaque (Base32 or Base64) string/token that can be used to resume a download.
.../s3-transfer-manager/src/main/java/software/amazon/awssdk/transfer/s3/S3TransferManager.java
Outdated
Show resolved
Hide resolved
...nager/src/main/java/software/amazon/awssdk/transfer/s3/exception/TransferPauseException.java
Outdated
Show resolved
Hide resolved
...r-manager/src/main/java/software/amazon/awssdk/transfer/s3/internal/DefaultFileDownload.java
Outdated
Show resolved
Hide resolved
.../src/main/java/software/amazon/awssdk/transfer/s3/internal/progress/DownloadFileMonitor.java
Outdated
Show resolved
Hide resolved
.../src/main/java/software/amazon/awssdk/transfer/s3/internal/progress/DownloadFileMonitor.java
Outdated
Show resolved
Hide resolved
.../main/java/software/amazon/awssdk/transfer/s3/internal/progress/TransferListenerContext.java
Outdated
Show resolved
Hide resolved
.../main/java/software/amazon/awssdk/transfer/s3/internal/progress/TransferProgressUpdater.java
Outdated
Show resolved
Hide resolved
| @@ -204,6 +205,8 @@ private Context() { | |||
| */ | |||
| TransferObjectRequest request(); | |||
|
|
|||
| SdkResponse initialResponse(); | |||
There was a problem hiding this comment.
Javadoc!
Also, can this be a S3Response?
| private final long bytesTransferred; | ||
| private final Instant lastModified; | ||
|
|
||
| private PersistableFileDownload(DefaultBuilder builder) { |
There was a problem hiding this comment.
Wondering if we should include a (perhaps optional) checksum of the bytes written so far, which a resume would validate before appending to an existing file. This would help ensure we don't clobber a file if it was modified by another source.
zoewangg
changed the base branch from
master
to
zoewang/tm-download-pause-resume
dagnir
reviewed
Mar 23, 2022
...r-manager/src/main/java/software/amazon/awssdk/transfer/s3/internal/DefaultFileDownload.java
Outdated
Show resolved
Hide resolved
...r-manager/src/main/java/software/amazon/awssdk/transfer/s3/internal/DefaultFileDownload.java
Show resolved
Hide resolved
1 task
zoewangg
changed the title
zoewangg
force-pushed
the
zoewang/downloadPauseResume
branch
from
3a576e2
to
9a4a68e
Compare
|
SonarCloud Quality Gate failed. |
12 tasks
zoewangg
added a commit
that referenced
this pull request
Apr 27, 2022
* [TM DownloadFile Pause and Resume] Part 1: Add configuration to enable overwriting existing files (#3125) * Expose an option to overwrite an existing file in FileAsyncResponseTransformer * Add changelog entries and make TM use CREATE_OR_REPLACE_EXISTING write option by default * Address feedback * Update and address feedback * [TM DownloadFile Pause and Resume] Part 2: Implement pause for downloadFile operation (#3094) * Part 1: Implement pause for downloadFile operation * Address feedback * Refactor the logic * Address feedback * Fix merging issue * [TM DownloadFile Pause and Resume] Part 3: Implement resumeDownloadFile (#3157) * Implement resumeDownloadFile * Move test code around * Address feedback * Fix flaky test * fix flaky integ test * add changelog entry * Troubleshooting flaky test * Add file length check when checking if file has modified or not * Address feedback
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
This is the part 2 of the implementation to support pause and resume for downloadFile operation
This PR will be merged to a feature branch once approved.
Modifications
This is still in draft.Add support for pause fordownloadFileoperationTesting
Added unit tests and integ tests
Screenshots (if appropriate)
Types of changes
Checklist
mvn installsucceedsscripts/new-changescript and following the instructions. Commit the new file created by the script in.changes/next-releasewith your changes.License