Pause/Resume Upload for Transfer Manager with Java S3Client #4886

davidh44 · 2024-02-03T00:08:00Z

Motivation and Context

Currently pausing/resuming a file upload with TransferManager is only supported with CRT S3 Client.

Modifications

Adding support when using Java S3 Client

Testing

added integ tests
to add unit tests in separate PR

Screenshots (if appropriate)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Checklist

I have read the CONTRIBUTING document
Local run of mvn install succeeds
My code follows the code style of this project
My change requires a change to the Javadoc documentation
I have updated the Javadoc documentation accordingly
I have added tests to cover my changes
All new and existing tests passed
I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

I confirm that this pull request can be released under the Apache 2 license

…atalog Settings. Introduce SASL/PLAIN as an authentication method for Glue Kafka connections

…edf75047e Pull request: release <- staging/96e1b8b2-3a37-418d-b480-f27edf75047e

This is a reintroduction of the reverted commit #4867. This includes a fix to the issue that caused the revert: improper handling of empty header values.

zoewangg · 2024-02-05T21:02:31Z

...va/software/amazon/awssdk/transfer/s3/S3TransferManagerUploadPauseResumeIntegrationTest.java

        resumedUpload.completionFuture().join();
        verifyMultipartUploadIdNotExist(resumableFileUpload);
        assertThat(resumedUpload.progress().snapshot().totalBytes()).hasValue(bytes.length);
+
+        Files.write(largeFile.toPath(), originalBytes);


Unintentional change?

we need to write back the original largeFile , since we overwrote it with "helloworld" to test file changed,
we are doing parameterized tests so this will run twice, one for each type of TM

Should we do it in afterEach?

we only overwrite it in this test, so dont need to write back for other tests

...ager/src/main/java/software/amazon/awssdk/transfer/s3/internal/GenericS3TransferManager.java

...m/s3-transfer-manager/src/main/java/software/amazon/awssdk/transfer/s3/model/FileUpload.java

.../s3/src/main/java/software/amazon/awssdk/services/s3/internal/multipart/PauseObservable.java

...oftware/amazon/awssdk/services/s3/internal/multipart/UploadWithKnownContentLengthHelper.java

* including S3 Access Grants Plugin as part of Java SDK Bundle --------- Co-authored-by: Shiva Kumar Mukkapati <mshvkmr@amazon.com>

- .json files files for each version are grouped under minor versions in .changes - Markdown files for each minor version are created in changelogs/ directory - Changelog scripts updated to add a link to older version in the generated changelog

…only update to address customer issues.

…or use with your CAPTCHA JavaScript integration API.

…to the customers on the changes that they make on the domain.

…hQL APIs

…pArn, to the response of the logs:DescribeLogGroups action.

…ty to the customers on the changes that they make on the domain.

…c59dc86fd Pull request: release <- staging/6f43c303-a28b-4966-b004-d38c59dc86fd

zoewangg · 2024-02-06T22:56:04Z

...va/software/amazon/awssdk/transfer/s3/S3TransferManagerUploadPauseResumeIntegrationTest.java

        resumedUpload.completionFuture().join();
        verifyMultipartUploadIdNotExist(resumableFileUpload);
        assertThat(resumedUpload.progress().snapshot().totalBytes()).hasValue(bytes.length);
+
+        Files.write(largeFile.toPath(), originalBytes);


Should we do it in afterEach?

...ager/src/main/java/software/amazon/awssdk/transfer/s3/internal/GenericS3TransferManager.java

...manager/src/main/java/software/amazon/awssdk/transfer/s3/internal/TransferManagerHelper.java

zoewangg · 2024-02-06T23:34:49Z

...oftware/amazon/awssdk/services/s3/internal/multipart/UploadWithKnownContentLengthHelper.java

+                                                   .uploadId(uploadId)
+                                                   .bucket(putObjectRequest.bucket())
+                                                   .key(putObjectRequest.key())
+                                                   .partNumberMarker(0)


Is there any other parameter we should pass? Should we add a convert method?

yup we need SSE-C fields, added conversion method to setAllFields

...oftware/amazon/awssdk/services/s3/internal/multipart/UploadWithKnownContentLengthHelper.java

zoewangg · 2024-02-06T23:38:44Z

...oftware/amazon/awssdk/services/s3/internal/multipart/UploadWithKnownContentLengthHelper.java

+                                                   .build();
+        ListPartsPublisher listPartsPublisher = s3AsyncClient.listPartsPaginator(request);
+        SdkPublisher<Part> partsPublisher = listPartsPublisher.parts();
+        partsPublisher.subscribe(part -> existingParts.put(part.partNumber(), SdkPojoConversionUtils.toCompletedPart(part)));


How can we guarantee this gets finished before uploadParts?

passing future and calling join() in Subscriber :: onNext()

zoewangg · 2024-02-07T00:11:38Z

.../s3/src/main/java/software/amazon/awssdk/services/s3/internal/multipart/PauseObservable.java

+@SdkInternalApi
+public class PauseObservable {
+
+    private volatile UploadWithKnownContentLengthHelper.KnownContentLengthAsyncRequestBodySubscriber subscriber;


To avoid exposing internal class and implementation detail, suggesting creating another layer of abstraction like this: hdavidh/pause-resume-upload-java...zoewang/PR4886-review

...es/s3/src/main/java/software/amazon/awssdk/services/s3/internal/multipart/S3ResumeToken.java

...oftware/amazon/awssdk/services/s3/internal/multipart/UploadWithKnownContentLengthHelper.java

…ng files or objects to transfer.

…ft Advisor recommendations.

…t replication feature as part of Lex Global Resiliency offering. This feature leverages a new set of APIs that allow customers to create bot replicas and replicate changes to bots across regions.

…2798c8cb1 Pull request: release <- staging/1c0c8ade-7bcb-4158-8160-4962798c8cb1

...3-transfer-manager/src/it/java/software/amazon/awssdk/transfer/s3/S3IntegrationTestBase.java

...fer-manager/src/main/java/software/amazon/awssdk/transfer/s3/internal/PauseResumeHelper.java

...s/s3/src/main/java/software/amazon/awssdk/services/s3/internal/multipart/PausibleUpload.java

...ager/src/main/java/software/amazon/awssdk/transfer/s3/internal/GenericS3TransferManager.java

… Chart Color Configuration; Documentation Update

…rallel & queued execution modes and add support for triggers with filtering on branches and file paths.

…ture. This feature allows Workspaces Core customers to provision workspaces without providing users. CreateWorkspaces and DescribeWorkspaces APIs will now take a new optional parameter "WorkspaceName".

…9f93cd5d9 Pull request: release <- staging/e358ef45-7969-4222-a98d-7fc9f93cd5d9

* Configure modeled signer properties for endpoint based auth scheme resolver * Refactor the logic to use Knowledge Indexes

* AWS Glue Update: Introduce Catalog Encryption Role within Glue Data Catalog Settings. Introduce SASL/PLAIN as an authentication method for Glue Kafka connections * Amazon WorkSpaces Update: Added definitions of various WorkSpace states * Release 2.23.18. Updated CHANGELOG.md, README.md and all pom.xml. * update aws-sdk-java pom to add imds and dyanmodb-enhanced (#4890) * Performance improvement for sigv4 signing. (#4891) This is a reintroduction of the reverted commit #4867. This includes a fix to the issue that caused the revert: improper handling of empty header values. * Update to next snapshot version: 2.23.19-SNAPSHOT * Delete CloudSearchv2IntegrationTest (#4888) * Fix tag deletion command (#4892) * including S3 Access Grants Plugin as part of Java SDK Bundle (#4881) * including S3 Access Grants Plugin as part of Java SDK Bundle --------- * Archive old changelog entries (< 2.23.0) (#4873) - .json files files for each version are grouped under minor versions in .changes - Markdown files for each minor version are created in changelogs/ directory - Changelog scripts updated to add a link to older version in the generated changelog * Changing indentation of config files to 4 spaces (#4889) * Amazon EC2 Container Service Update: This release is a documentation only update to address customer issues. * AWS WAFV2 Update: You can now delete an API key that you've created for use with your CAPTCHA JavaScript integration API. * Amazon OpenSearch Service Update: This release adds clear visibility to the customers on the changes that they make on the domain. * AWS AppSync Update: Support for environment variables in AppSync GraphQL APIs * Amazon CloudWatch Logs Update: This release adds a new field, logGroupArn, to the response of the logs:DescribeLogGroups action. * Amazon Elasticsearch Service Update: This release adds clear visibility to the customers on the changes that they make on the domain. * Release 2.23.19. Updated CHANGELOG.md, README.md and all pom.xml. * Update to next snapshot version: 2.23.20-SNAPSHOT * Bump CRT version and expose setting memory limits for S3 calls (#4885) * Bump aws-crt version to 0.29.9 * Exposes a setting to set the memory limit when making asynchronous calls with the CRT-based S3 client * Activating SRA for this service (#4896) * Fix request cancellation logic in the AWS CRT Sync HTTP client (#4887) * Fix request cancellation logic in the AWS CRT Sync HTTP client * Address feedback * Fix bug to allow MpuS3Client to PUT COPY with SSE-C and Checksum * Enable CRC32 for Multipart PUT COPY * Address comments * add changelog and update checksum check * Address comments * update javadocs * Add unit tests --------- Co-authored-by: AWS <> Co-authored-by: aws-sdk-java-automation <43143862+aws-sdk-java-automation@users.noreply.github.com> Co-authored-by: Zoe Wang <33073555+zoewangg@users.noreply.github.com> Co-authored-by: Matthew Miller <millem@amazon.com> Co-authored-by: John Viegas <70235430+joviegas@users.noreply.github.com> Co-authored-by: Dongie Agnir <261310+dagnir@users.noreply.github.com> Co-authored-by: shiva kumar <62441208+shiva958@users.noreply.github.com> Co-authored-by: Shiva Kumar Mukkapati <mshvkmr@amazon.com> Co-authored-by: Anna-Karin Salander <salande@amazon.com>

* Fix bug to allow MpuS3Client to PUT COPY with SSE-C and Checksum * Enable CRC32 for Multipart PUT COPY * Address comments * add changelog and update checksum check * Address comments * update javadocs * Add unit tests

…4905)" This reverts commit 1849320.

…pause-resume-upload-java

sonarcloud · 2024-02-09T00:49:35Z

Quality Gate failed

Failed conditions
57.6% Coverage on New Code (required ≥ 80%)
3.2% Duplication on New Code (required ≤ 3%)
B Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

davidh44 · 2024-02-09T01:41:13Z

messy merges from other PR, closing in favor of new PR
#4908

Pause/Resume Upload for Transfer Manager with Java S3Client

745e402

davidh44 requested a review from a team as a code owner February 3, 2024 00:08

davidh44 and others added 11 commits February 2, 2024 17:15

Fix checkstyle and equalsVerifier

65069fb

temporarily disable failing test

3dfa1c2

AWS Glue Update: Introduce Catalog Encryption Role within Glue Data C…

2fa0d9e

…atalog Settings. Introduce SASL/PLAIN as an authentication method for Glue Kafka connections

Amazon WorkSpaces Update: Added definitions of various WorkSpace states

4d17804

Release 2.23.18. Updated CHANGELOG.md, README.md and all pom.xml.

fd010e4

Merge pull request #2890 from aws/staging/96e1b8b2-3a37-418d-b480-f27…

069a00c

…edf75047e Pull request: release <- staging/96e1b8b2-3a37-418d-b480-f27edf75047e

update aws-sdk-java pom to add imds and dyanmodb-enhanced (#4890)

9aeba32

Performance improvement for sigv4 signing. (#4891)

0fedea3

This is a reintroduction of the reverted commit #4867. This includes a fix to the issue that caused the revert: improper handling of empty header values.

Merge remote-tracking branch 'public/release'

1089532

Update to next snapshot version: 2.23.19-SNAPSHOT

fce9bf9

Delete CloudSearchv2IntegrationTest (#4888)

1d99321

zoewangg reviewed Feb 5, 2024

View reviewed changes

dagnir and others added 14 commits February 5, 2024 15:34

Fix tag deletion command (#4892)

6afc377

including S3 Access Grants Plugin as part of Java SDK Bundle (#4881)

007ee65

* including S3 Access Grants Plugin as part of Java SDK Bundle --------- Co-authored-by: Shiva Kumar Mukkapati <mshvkmr@amazon.com>

Address comments

7a3589c

Changing indentation of config files to 4 spaces (#4889)

f323822

Amazon EC2 Container Service Update: This release is a documentation …

d10ddf6

…only update to address customer issues.

AWS WAFV2 Update: You can now delete an API key that you've created f…

cd207db

…or use with your CAPTCHA JavaScript integration API.

Amazon OpenSearch Service Update: This release adds clear visibility …

ca9b69b

…to the customers on the changes that they make on the domain.

AWS AppSync Update: Support for environment variables in AppSync Grap…

f92e9f3

…hQL APIs

Amazon CloudWatch Logs Update: This release adds a new field, logGrou…

4275763

…pArn, to the response of the logs:DescribeLogGroups action.

Amazon Elasticsearch Service Update: This release adds clear visibili…

b020266

…ty to the customers on the changes that they make on the domain.

Release 2.23.19. Updated CHANGELOG.md, README.md and all pom.xml.

9597393

Merge pull request #2891 from aws/staging/6f43c303-a28b-4966-b004-d38…

9ab7d51

…c59dc86fd Pull request: release <- staging/6f43c303-a28b-4966-b004-d38c59dc86fd

Update to next snapshot version: 2.23.20-SNAPSHOT

2336fed

zoewangg reviewed Feb 7, 2024

View reviewed changes

...oftware/amazon/awssdk/services/s3/internal/multipart/UploadWithKnownContentLengthHelper.java Outdated Show resolved Hide resolved

...oftware/amazon/awssdk/services/s3/internal/multipart/UploadWithKnownContentLengthHelper.java Outdated Show resolved Hide resolved

AWS and others added 10 commits February 7, 2024 19:06

AWS DataSync Update: AWS DataSync now supports manifests for specifyi…

6a83122

…ng files or objects to transfer.

Amazon Redshift Update: LisRecommendations API to fetch Amazon Redshi…

7ab1028

…ft Advisor recommendations.

Amazon Lex Model Building V2 Update: This release introduces a new bo…

e16376e

…t replication feature as part of Lex Global Resiliency offering. This feature leverages a new set of APIs that allow customers to create bot replicas and replicate changes to bots across regions.

Updated endpoints.json and partitions.json.

9d8d4cf

Release 2.23.20. Updated CHANGELOG.md, README.md and all pom.xml.

d182576

Merge pull request #2892 from aws/staging/1c0c8ade-7bcb-4158-8160-496…

0c2d4eb

…2798c8cb1 Pull request: release <- staging/1c0c8ade-7bcb-4158-8160-4962798c8cb1

Update to next snapshot version: 2.23.21-SNAPSHOT

4a81f88

Address comments

a5b277b

Wrap subscriber with PausibleUpload

4bc0a09

add changelog

a963bdf

cenedhryn reviewed Feb 8, 2024

View reviewed changes

...ager/src/main/java/software/amazon/awssdk/transfer/s3/internal/GenericS3TransferManager.java Show resolved Hide resolved

AWS and others added 13 commits February 8, 2024 19:04

Amazon QuickSight Update: General Interactions for Visuals; Waterfall…

02e807c

… Chart Color Configuration; Documentation Update

AWS CodePipeline Update: Add ability to execute pipelines with new pa…

ee01442

…rallel & queued execution modes and add support for triggers with filtering on branches and file paths.

Amazon WorkSpaces Update: This release introduces User-Decoupling fea…

ab6113b

…ture. This feature allows Workspaces Core customers to provision workspaces without providing users. CreateWorkspaces and DescribeWorkspaces APIs will now take a new optional parameter "WorkspaceName".

Release 2.23.21. Updated CHANGELOG.md, README.md and all pom.xml.

4ef752f

Merge pull request #2893 from aws/staging/e358ef45-7969-4222-a98d-7fc…

1548dd5

…9f93cd5d9 Pull request: release <- staging/e358ef45-7969-4222-a98d-7fc9f93cd5d9

Update to next snapshot version: 2.23.22-SNAPSHOT

f3da726

S3 should configure signer properties (#4856)

2f2e6e1

* Configure modeled signer properties for endpoint based auth scheme resolver * Refactor the logic to use Knowledge Indexes

integ test upload resume with different TMs

207a10e

Rename to PausableUpload

824d6ad

Enable CRC32 for PUT for MultipartS3AsyncClient (#4898)

6411744

* Fix bug to allow MpuS3Client to PUT COPY with SSE-C and Checksum * Enable CRC32 for Multipart PUT COPY * Address comments * add changelog and update checksum check * Address comments * update javadocs * Add unit tests

Revert "Merge Enable Crc32 change into Pause-Resume Branch (#4904) (#…

fcfc6a5

…4905)" This reverts commit 1849320.

Merge commit '6411744f8e3e9c3b11a1383921ad4b8e4f822aee' into hdavidh/…

6bc4130

…pause-resume-upload-java

davidh44 mentioned this pull request Feb 9, 2024

Pause/Resume Upload for Transfer Manager with Java S3Client #4908

Merged

12 tasks

davidh44 closed this Feb 9, 2024

davidh44 deleted the hdavidh/pause-resume-upload-java branch February 9, 2024 01:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pause/Resume Upload for Transfer Manager with Java S3Client #4886

Pause/Resume Upload for Transfer Manager with Java S3Client #4886

davidh44 commented Feb 3, 2024

zoewangg Feb 5, 2024

davidh44 Feb 5, 2024

zoewangg Feb 6, 2024

davidh44 Feb 7, 2024

zoewangg Feb 6, 2024

zoewangg Feb 6, 2024

davidh44 Feb 7, 2024

zoewangg Feb 6, 2024

davidh44 Feb 7, 2024

zoewangg Feb 7, 2024

sonarcloud bot commented Feb 9, 2024

davidh44 commented Feb 9, 2024

Pause/Resume Upload for Transfer Manager with Java S3Client #4886

Pause/Resume Upload for Transfer Manager with Java S3Client #4886

Conversation

davidh44 commented Feb 3, 2024

Motivation and Context

Modifications

Testing

Screenshots (if appropriate)

Types of changes

Checklist

License

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Feb 9, 2024

Quality Gate failed

davidh44 commented Feb 9, 2024