S3 default integrity protection change - version 2.30.0

[debora-ito]

In AWS SDK for Java 2.30.0, we released changes to the S3 client that adopts new default integrity protections. For more information on default integrity behavior, please refer to the official SDK documentation. In SDK releases from this version on, clients default to enabling an additional checksum on all Put calls and enabling validation on Get calls.

You can disable default integrity protections for S3. We do not recommend this because checksums are important to S3 integrity posture. Integrity protections can be disabled by setting the config flag to WHEN_REQUIRED, or by using the related AWS shared config file settings or environment variables.

Disclaimer: the AWS SDKs and CLI are designed for usage with official AWS services. We may introduce and enable new features by default, such as these new default integrity protections, prior to them being supported or otherwise handled by third-party service implementations. You can disable the new behavior with the WHEN_REQUIRED value for the request_checksum_calculation and response_checksum_validation configuration options covered in Data Integrity Protections for Amazon S3.

[steveloughran]

What's the measured performance hit of the new option? If the cost is negligible then I'm happy -but since we've been burned by the v2 default of checksum validation tangibly hurting benchmarks, we now set .checksumValidationEnabled(false) for S3 clients by default.

As these are only CRCs they should be very low cost -correct?

[joviegas]

Yeap you are correct this should be low cost.
Please refer our Bench mark test details in #5701

[zoewangg]

To add to what John mentioned, checksum calculation was enabled by default using MD5 in Java SDK even before 2.30.0, as part of this change, we switched to CRC32 from MD5, which means the performance should be better, if anything.

[steveloughran]

Moot until we can work out how to turn it off completely so work with external stores.

[joviegas]

Workaround to skip default checksum calculations for operations that don't require them and add MD5 checksums for operations that require checksums

1. Skip Default Checksum Calculation for operation which do not require checksum

Choose one of these options:

Option A: Add to AWS profile configuration

[YOUR_DEFAULT_PROFILE]
request_checksum_calculation = when_required

Or

Option B: Set system property

import software.amazon.awssdk.core.SdkSystemSetting;
import software.amazon.awssdk.services.s3.S3Client;

public class DemoClass {

    public static void main(String[] args) {
    
        System.setProperty(SdkSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION.property(), 
                                                                        "WHEN_REQUIRED");

        S3Client s3Client = S3Client.builder().build();

        s3Client.listBuckets();
    }

Adding MD5 checksum for operations which require checksum calculation

Step 1: Add Interceptor class as below

import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.Optional;

import software.amazon.awssdk.checksums.DefaultChecksumAlgorithm;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.core.checksums.ChecksumSpecs;
import software.amazon.awssdk.core.interceptor.Context;
import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
import software.amazon.awssdk.core.interceptor.ExecutionInterceptor;
import software.amazon.awssdk.core.interceptor.SdkExecutionAttribute;
import software.amazon.awssdk.core.interceptor.SdkInternalExecutionAttribute;
import software.amazon.awssdk.core.interceptor.trait.HttpChecksum;
import software.amazon.awssdk.core.internal.util.HttpChecksumUtils;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.http.Header;
import software.amazon.awssdk.http.SdkHttpRequest;
import software.amazon.awssdk.utils.Md5Utils;

public class Md5RequiredOperationInterceptor implements ExecutionInterceptor {

    @Override
    public SdkHttpRequest modifyHttpRequest(Context.ModifyHttpRequest context, ExecutionAttributes executionAttributes) {
        boolean isHttpChecksumRequired = isHttpChecksumRequired(executionAttributes);
        boolean requestAlreadyHasMd5 = context.httpRequest().firstMatchingHeader(Header.CONTENT_MD5).isPresent();

        Optional<RequestBody> syncContent = context.requestBody();
        Optional<AsyncRequestBody> asyncContent = context.asyncRequestBody();

        if (!isHttpChecksumRequired || requestAlreadyHasMd5) {
            return context.httpRequest();
        }

        if (asyncContent.isPresent()) {
            throw new IllegalStateException("This operation requires a content-MD5 checksum, " +
                    "but one cannot be calculated for non-blocking content.");
        }

        if (syncContent.isPresent()) {
            try {
                String payloadMd5 = Md5Utils.md5AsBase64(syncContent.get().contentStreamProvider().newStream());
                return context.httpRequest().copy(r -> r.putHeader(Header.CONTENT_MD5, payloadMd5));
            } catch (IOException e) {
                throw new UncheckedIOException(e);
            }
        }
        return context.httpRequest();
    }

    private boolean isHttpChecksumRequired(ExecutionAttributes executionAttributes) {
        return executionAttributes.getAttribute(SdkInternalExecutionAttribute.HTTP_CHECKSUM_REQUIRED) != null
                || isMd5ChecksumRequired(executionAttributes);
    }

    public static boolean isMd5ChecksumRequired(ExecutionAttributes executionAttributes) {
        ChecksumSpecs resolvedChecksumSpecs = getResolvedChecksumSpecs(executionAttributes);
        if (resolvedChecksumSpecs == null) {
            return false;
        } else {
            return resolvedChecksumSpecs.algorithm() == null && resolvedChecksumSpecs.isRequestChecksumRequired();
        }
    }

    public static ChecksumSpecs getResolvedChecksumSpecs(ExecutionAttributes executionAttributes) {
        ChecksumSpecs checksumSpecs = executionAttributes.getAttribute(SdkExecutionAttribute.RESOLVED_CHECKSUM_SPECS);
        return checksumSpecs != null ? checksumSpecs : resolveChecksumSpecs(executionAttributes);
    }

    public static ChecksumSpecs resolveChecksumSpecs(ExecutionAttributes executionAttributes) {
        HttpChecksum httpChecksumTraitInOperation = executionAttributes.getAttribute(SdkInternalExecutionAttribute.HTTP_CHECKSUM);
        if (httpChecksumTraitInOperation == null) {
            return null;
        } else {
            boolean hasRequestValidation = httpChecksumTraitInOperation.requestValidationMode() != null;
            String requestAlgorithm = httpChecksumTraitInOperation.requestAlgorithm();
            String checksumHeaderName = requestAlgorithm != null ? HttpChecksumUtils.httpChecksumHeader(requestAlgorithm) : null;
            return ChecksumSpecs.builder().algorithmV2(DefaultChecksumAlgorithm.fromValue(requestAlgorithm)).headerName(checksumHeaderName).responseValidationAlgorithmsV2(httpChecksumTraitInOperation.responseAlgorithmsV2()).isValidationEnabled(hasRequestValidation).isRequestChecksumRequired(httpChecksumTraitInOperation.isRequestChecksumRequired()).isRequestStreaming(httpChecksumTraitInOperation.isRequestStreaming()).requestAlgorithmHeader(httpChecksumTraitInOperation.requestAlgorithmHeader()).build();
        }
    }
}

Step 2: Use this interceptor in the client builder as below

import software.amazon.awssdk.services.s3.S3Client;

public class DemoChecksumRollback {

    public static void main(String[] args) {

        Md5RequiredOperationInterceptor md5RequiredOperationInterceptor =
                new Md5RequiredOperationInterceptor();

        S3Client s3Client = S3Client.builder()
                .overrideConfiguration(override ->
                        override.addExecutionInterceptor(md5RequiredOperationInterceptor))
                .build();

        s3Client.listBuckets();
    }
}

3. Doing Both: skipping Checksum for operations that do not require checksum and adding MD5 checksum for operations that require checksum

Add profile configuration as mentioned above or set System Property as below. Add Md5RequiredOperationInterceptor as below

import software.amazon.awssdk.services.s3.S3Client;

public class DemoChecksumRollback {
    public static void main(String[] args) {

        System.setProperty(SdkSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION.property(), "WHEN_REQUIRED");
        // Md5RequiredOperationInterceptor is same as class defined in above section
        Md5RequiredOperationInterceptor md5RequiredOperationInterceptor = 
                new Md5RequiredOperationInterceptor();

        S3Client s3Client = S3Client.builder()
                .overrideConfiguration(override ->
                        override.addExecutionInterceptor(md5RequiredOperationInterceptor))
                .build();

        s3Client.listBuckets();
    }
}

Note: The above workarounds will not prevent adding Default checksums for operations which require checksum like DeleteObjects etc.

[UdayBidikar]

Would be nice if AWS puts MD5 checksum also under the control of a new system option!

[joviegas]

Would be nice if AWS puts MD5 checksum also under the control of a new system option!

Thank you for the feedback. Your suggestions have been noted and forwarded to the appropriate internal teams.

[HeyImAllan]

We are having this issue with Dell ECS. Thanks a lot for this! You are doing amazing work here.

[steveloughran]

Is there a single switch in the client builder to say "no request checksums?"

As that is all we should need to restore behaviour such that the SDK will work with third pary stores again?

Everyone seems to

• iceberg: do it for delete/put requests: S3: Add LegacyMd5Plugin to S3 client builder apache/iceberg#12264
• trinio: do it for delete requests: Disable strong integrity checks for S3 FS  trinodb/trino#24954

these both force in a different signer for these requests.

Is that the only way to do it? I really want to decouple checksums from the signing, as our code supports people providing their own signers, and I don't want their code to break from us adding a workaround to what is effectively a regression "work with other stores" in the SDK".

[zoewangg]

There is no single switch to disable checksums altogether; you can only disable checksums for operations that do not require checksums such as GetObject and PutObject, and it will always be enabled for operations that require it (as defined in service model).

[steveloughran]

I see there is now some stuff in the SdkClientBuilder to set the request calculation via an RequestChecksumCalculation enum.

• is this documented?
• does it do the right thing: set checksums when needed, not break all interaction with third party stores otherwise.

Be aware that I've given up trying to get the Hadoop s3a connector onto a 2.30.x release; it'll be stuck at 2.29.52. This is a shame as well as some transient CVEs to worry about, I did want that fix for IAM credential throttling we'd seen.

If it is fixable just by that builder option then we can pick up a more recent version.

[zoewangg]

It's documented here https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html You can configure this behavior through different places, system property, env var, client config, etc.

Yeah, sounds like RequestChecksumCalculation.WHEN_REQUIRED is what you are looking for, which only enables checksum calculation (CRC32 by default) for operations that require it.

Note that it may break third party stores that only accept MD5 though since the SDK no longer sends MD5 for those operations, and you'd need to follow the workaround provided above to add MD5

[steveloughran]

@joviegas

You can configure this behaviour through different places, system property, env var, client config, etc.

It's not documented on that page; I had actually checked that before I made my previous comment. I can see it in the SDK, which is why I would like a formal documentation of it so that I can be reasonably confident it will not be removed and that its semantics will remain consistent over time. Unless you can show me where I missed it -please can you update your documentation.

I have frequently expressed my unhappiness with certain aspects of the V2 SDK. Most of my issues that have been related to its of handling failure conditions -the rare ones which inevitably do not show up during QE but yet do with the petabytes of data read and written through the S3A connector every day. That's why I see issues #5415 and #3846 end up surfacing before anyone else notices. I think we are still identifying those failure conditions where reporting them to caller is different from the V1 SDK HADOOP-19347. S3A: AWS SDK deleteObjects() doesn't handle 500 failures of individual objects. Then of course there are the changes which appear to be minor, but through their repeated appearance end up delaying releases until we have a workaround, e.g. HADOOP-19272. S3A: AWS SDK 2.25.53 warnings logged about transfer manager not using CRT client

I consider making this library incompatible with third-party stores a regression. I see you provide code which claims to be workaround -but if I see no evidence that it does work and given your demonstrable lack of testing with external stores -how can I trust it? Furthermore -how can I trust the next minor release of the SDK not to break things again?

Please restore compatibility without requiring every single user of your SDK to either add extra code or only belatedly discover this incompatible change in production.

I'm leaving HADOOP-19490. S3A: AWS SDK 2.30+ incompatible with third party stores for someone other than myself to address. I estimate that I spent about 10 to 12 weeks last year dealing with SDK v2 issues and I'm not doing anything this year if blocking updates is a valid solution.

I did actually want this upgrade as the fix to the IAM throttling failure of #3846; we are the only people who have encountered this and it is not in a previous release. It also means that any of your colleagues do you want to get any new feature into the SDK or downstream applications -they get to do all the homework of fixing and testing it.

Can I close by proposing that you schedule an AWS event session on your AWSEvents Youtube channel? It might be a better way than adding comments in this issue.

cc: @ahmarsuhail shameersss1

[joviegas]

Hi @steveloughran
I'm sorry for the challenges you've faced with the V2 SDK and appreciate your detailed feedback.
I've tested the workaround by installing Minio version RELEASE.2025-01-18T00-31-37Z (before Minio's fix). Before implementing the workaround, we encountered the error; after applying it, the requests succeeded without issues.

We will update the developer guide for checksums to include documentation on the environment variable and system property options to disable checksums.

Regarding the AWS Events YouTube channel session suggestion, I've communicated this to the appropriate team for consideration.

We will also sync up with @ahmarsuhail to work together with you to collect and address your pain points with the SDK.

[uwolfer]

Just for the record, I also run into this issue while using this sdk for connecting against Google cloud storage: #5987

Would be nice if you could improve 3rd party S3 support in the future again - it used to be a nice library which just worked flawlessly. Or at least add a big warning to the API when trying to use non-AWS S3 endpoints.