Cannot upload large files using Multipart to an S3 bucket with both Object Locking and Encryption Enabled.

[jfleming-ic]

Describe the bug

We are attempting to upload large files to an object locked bucket that has encryption enabled. We use multipart uploads and encryption with an object locked bucket. We set the following configuration on the transferManager

setAlwaysCalculateMultipartMd5(true)

However we receive the following stacktrace when attempting to upload:

com.amazonaws.services.s3.model.AmazonS3Exception: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Part requests with Object Lock parameters (Service: Amazon S3; Status Code: 400; Error Code: InvalidRequest; Request ID:; S3 Extended Request ID:; Proxy: null) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3887) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3872) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadPartsInSeries(UploadCallable.java:323) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInParts(UploadCallable.java:226) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:147) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:115) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:45) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) Sep 19 01:30:35 ip-10-66-172-138 cassandra-backup[77160]: at java.base/java.lang.Thread.run(Thread.java:829)

Expected Behavior

Uploads should respect the following configuration when uploading parts in parallel when encryption is enabled.

transferManager.getConfiguration().setAlwaysCalculateMultipartMd5(true)

Uploads using multi part should work on an S3 bucket with both encryption and object locking.

Current Behavior

We are required to increase our multipart thresh hold to prevent multipart upload from being executed, which causes a steep performance penalty on large uploads

Reproduction Steps

1 - Create an Object locked S3 bucket with encryption enabled.
2 - Create a file with 50MB of content at /my-mega-file.txt.
3 - The following code should trigger the issue (I don't think I can share our actual production code that's hitting this, so this is a rough mockup of it)

public static void main(String[] args) throws IOException {

    String bucketName = "my-bucket-for-aws";
    String keyName = "my-file";
    File file = new File("/my-mega-file.txt");
    String kmsKeyId = "some-key-id";

    AWSKMS kmsClient = AWSKMSClientBuilder.standard()
            .withRegion(Regions.DEFAULT_REGION)
            .build();

    AmazonS3EncryptionV2 s3Encryption = AmazonS3EncryptionClientV2Builder.standard()
            .withRegion(Regions.DEFAULT_REGION)
            .withKmsClient(kmsClient)
            .withCryptoConfiguration(new CryptoConfigurationV2().withCryptoMode((CryptoMode.AuthenticatedEncryption)))
            .withEncryptionMaterialsProvider(new KMSEncryptionMaterialsProvider(kmsKeyId))
            .build();

    TransferManager tm = TransferManagerBuilder.standard()
            .withS3Client(s3Encryption)
            .withAlwaysCalculateMultipartMd5(true)
            .build();

    Upload upload = tm.upload(bucketName, keyName, file);

    try {
        upload.waitForCompletion();
    } catch (AmazonClientException | InterruptedException e) {
        System.out.println("Uh oh, this could be a bug");
    }
}

Possible Solution

When increasing the threshold size, we see the file upload successfully, however this is non-ideal in our use case.

Additional Information/Context

We did a bit of a dig around in the AWS SDK code base and found the following.

AWS checks the setAlwaysCalculateMultipartMd5 flag here

aws-sdk-java/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/internal/UploadCallable.java

Line 365 in 81b400b

futures.add(threadPool.submit(new UploadPartCallable(s3, request, shouldCalculatePartMd5())));

while performing a multi-part upload

However, that execution path is taken if uploading parts in parallel

aws-sdk-java/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/internal/UploadCallable.java

Line 223 in 81b400b

uploadPartsInParallel(requestFactory, uploadId);

Unfortunately, because we use encryption, parallel part upload is not supported and uploadPartsInSeries()

aws-sdk-java/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/internal/UploadCallable.java

Line 226 in 81b400b

return uploadPartsInSeries(requestFactory, uploadId);

gets executed. Sadly, that path doesn’t compute/include the MD5 hash and hence the error

AWS Java SDK version used

1.12.279

JDK version used

openjdk version "1.8.0_292"

Operating System and version

Debian 9

[jfleming-ic]

Hey any updates on this one? Any potential workarounds we could use in the short term?

[yasminetalby]

Hello @jfleming-ic ,

My apologies for the delay of answer on this issue.

We currently do not support setting Content MD5 with S3 Encryption Client for multipart-upload in both AWS Java SDK v1 and v2.

There has been similar submission in the v1 (#1936, #2052) repository in the past however this feature-request has been closed in favor of v2 feature implementation as the team is now concentrating its development effort on the AWS Java SDK v2.
There has been a similar feature request within the v2 repository (see). You can showcase interest on this feature by adding a thumbs up to the submission.

Unfortunately, there is no current workaround to this specific use case in both AWS Java SDK v1 and v2. In AWS Java SDK v1, you can only use multipart-upload with Object Lock parameters and Content MD5 with a regular, non-encryption S3 Client.

I apologies for any inconvenience.

Sincerely,

Yasmine

[jfleming-ic]

Thank you for the response @yasminetalby

I note that feature for SDK v2 is still open and doesn't look to be scheduled in (from what I can see), are there any timelines for this feature in V2 of the SDK?

[yasminetalby]

Of course!

Unfortunately we do not currently have publicly available timelines for feature-request implementation.
We use upvote counts and comments to help prioritize non-parity opened feature-request based on community feedback, interest and use cases. Once a feature has been implemented, community member who have interacted with the GitHub submission are notified as we update and close the issue submission.

Best regards,

Yasmine

[jfleming-ic]

Thanks! I'll close this issue off.

[github-actions]

COMMENT VISIBILITY WARNING

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.