Skip to content

cloud/amazon: fix AWS_SKIP_CHECKSUM not suppressing CRC32 on multipart uploads#166164

Merged
trunk-io[bot] merged 1 commit intocockroachdb:masterfrom
cockroach-teamcity:fix/issue-166090
Mar 19, 2026
Merged

cloud/amazon: fix AWS_SKIP_CHECKSUM not suppressing CRC32 on multipart uploads#166164
trunk-io[bot] merged 1 commit intocockroachdb:masterfrom
cockroach-teamcity:fix/issue-166090

Conversation

@cockroach-teamcity
Copy link
Member

After the aws-sdk-go-v2 s3 manager upgrade to v1.17.65, the manager's
internal initChecksumAlgorithm() unconditionally defaults ChecksumAlgorithm
to CRC32 when the field is empty. This means setting ChecksumAlgorithm = ""
on the PutObjectInput (which the code did when AWS_SKIP_CHECKSUM=true) no
longer suppresses checksums for multipart uploads — the manager overwrites
the cleared value with CRC32 before the S3 client sees it, bypassing
the RequestChecksumCalculation = WhenRequired configuration.

Fix this by adding smithy-go Initialize middleware via the Uploader's
ClientOptions that clears ChecksumAlgorithm from CreateMultipartUploadInput,
UploadPartInput, CompleteMultipartUploadInput, and PutObjectInput before
the S3 client's checksum middleware processes them. This runs after the
manager's initChecksumAlgorithm but before the client computes checksums,
effectively suppressing all checksum computation on multipart uploads when
AWS_SKIP_CHECKSUM=true.

Resolves: #166090

Release note (bug fix): Fixed a bug where the AWS_SKIP_CHECKSUM=true
storage option did not fully suppress checksums on multipart uploads
to S3-compatible storage. After an internal SDK upgrade, multipart
uploads included CRC32 checksums despite the skip flag, which could
cause errors with S3-compatible services that do not support CRC
checksums.

Generated by Claude Code Auto-Solver
Co-Authored-By: Claude noreply@anthropic.com


 pkg/cloud/amazon/BUILD.bazel        |  4 +++
 pkg/cloud/amazon/s3_storage.go      | 29 ++++++++++++++++++
 pkg/cloud/amazon/s3_storage_test.go | 59 +++++++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+)

This PR was auto-generated by issue-autosolve using Claude Code.
Please review carefully before approving.

@cockroach-teamcity cockroach-teamcity added the O-autosolver PRs originating from the autosolver bot label Mar 19, 2026
@cockroach-teamcity cockroach-teamcity requested a review from dt March 19, 2026 13:38
@trunk-io
Copy link
Contributor

trunk-io bot commented Mar 19, 2026

😎 Merged successfully - details.

@cockroach-teamcity
Copy link
Member Author

This change is Reviewable

func TestAddClearChecksumMiddleware(t *testing.T) {
defer leaktest.AfterTest(t)()

putInput := &s3svc.PutObjectInput{ChecksumAlgorithm: types.ChecksumAlgorithmCrc32}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are each only referenced once in the testcase literals so let's just inline them there.

@rafiss
Copy link
Collaborator

rafiss commented Mar 19, 2026

autosolver: in pkg/cloud/amazon/s3_storage_test.go, theputInput and similar variables are each only referenced once in the testcase literals so let's just inline them there.

…t uploads

After the aws-sdk-go-v2 s3 manager upgrade to v1.17.65, the manager's
internal initChecksumAlgorithm() unconditionally defaults ChecksumAlgorithm
to CRC32 when the field is empty. This means setting ChecksumAlgorithm = ""
on the PutObjectInput (which the code did when AWS_SKIP_CHECKSUM=true) no
longer suppresses checksums for multipart uploads — the manager overwrites
the cleared value with CRC32 before the S3 client sees it, bypassing
the RequestChecksumCalculation = WhenRequired configuration.

Fix this by adding smithy-go Initialize middleware via the Uploader's
ClientOptions that clears ChecksumAlgorithm from CreateMultipartUploadInput,
UploadPartInput, CompleteMultipartUploadInput, and PutObjectInput before
the S3 client's checksum middleware processes them. This runs after the
manager's initChecksumAlgorithm but before the client computes checksums,
effectively suppressing all checksum computation on multipart uploads when
AWS_SKIP_CHECKSUM=true.

Resolves: cockroachdb#166090

Release note (bug fix): Fixed a bug where the AWS_SKIP_CHECKSUM=true
storage option did not fully suppress checksums on multipart uploads
to S3-compatible storage. After an internal SDK upgrade, multipart
uploads included CRC32 checksums despite the skip flag, which could
cause errors with S3-compatible services that do not support CRC
checksums.

Generated by Claude Code Auto-Solver
Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions
Copy link
Contributor

[autosolve-response]

I've addressed the review comments and pushed updates.

Changes made:

Inlined the `putInput`, `uploadPartInput`, `createMultipartInput`, and `completeMultipartInput` variables directly into the test case table literals in `TestAddClearChecksumMiddleware`, since each was only referenced once. Test passes.

CHANGES_RESULT - SUCCESS

Please review the updated code.

Workflow run

@dt dt added backport-25.2.x Flags PRs that need to be backported to 25.2 backport-25.3.x Flags PRs that need to be backported to 25.3 backport-25.4.x Flags PRs that need to be backported to 25.4 backport-26.1.x Flags PRs that need to be backported to 26.1 labels Mar 19, 2026
@dt
Copy link
Contributor

dt commented Mar 19, 2026

/trunk merge

@github-actions
Copy link
Contributor

[autosolve-response]

I reviewed the comments but no code changes were necessary.

Analysis:

Test passes. The review feedback from `dt` was to inline variables that are only referenced once in the test case literals. Looking at the current code, this feedback has already been addressed — the inputs are already inlined directly into the test case struct literals (lines 927-930). No additional code changes are needed.

The `/trunk merge` issue comment is a merge command, not review feedback requiring code changes.

CHANGES_RESULT - SUCCESS

Workflow run

@trunk-io trunk-io bot merged commit f40aa74 into cockroachdb:master Mar 19, 2026
29 of 30 checks passed
@blathers-crl
Copy link

blathers-crl bot commented Mar 19, 2026

⚠️ End-of-Life Version Warning

You have requested backports to the following End-of-Life (EOL) versions:

  • 25.3

While Blathers will still create these backport PRs, please verify that backporting to EOL versions is intentional and appropriate. EOL versions no longer receive maintenance updates according to our support policy.

The backport PRs will also include this warning.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl
Copy link

blathers-crl bot commented Mar 19, 2026

Based on the specified backports for this PR, I applied new labels to the following linked issue(s). Please adjust the labels as needed to match the branches actually affected by the issue(s), including adding any known older branches.


Issue #166090: branch-release-26.1.


🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-25.2.x Flags PRs that need to be backported to 25.2 backport-25.3.x Flags PRs that need to be backported to 25.3 backport-25.4.x Flags PRs that need to be backported to 25.4 backport-26.1.x Flags PRs that need to be backported to 26.1 O-autosolver PRs originating from the autosolver bot target-release-25.4.8 target-release-26.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cloud/amazon: AWS_SKIP_CHECKSUM does not suppress crc checksum on multipart uploads

3 participants