cloud/amazon: fix AWS_SKIP_CHECKSUM not suppressing CRC32 on multipart uploads#166164
Conversation
|
😎 Merged successfully - details. |
pkg/cloud/amazon/s3_storage_test.go
Outdated
| func TestAddClearChecksumMiddleware(t *testing.T) { | ||
| defer leaktest.AfterTest(t)() | ||
|
|
||
| putInput := &s3svc.PutObjectInput{ChecksumAlgorithm: types.ChecksumAlgorithmCrc32} |
There was a problem hiding this comment.
These are each only referenced once in the testcase literals so let's just inline them there.
|
autosolver: in pkg/cloud/amazon/s3_storage_test.go, the |
…t uploads After the aws-sdk-go-v2 s3 manager upgrade to v1.17.65, the manager's internal initChecksumAlgorithm() unconditionally defaults ChecksumAlgorithm to CRC32 when the field is empty. This means setting ChecksumAlgorithm = "" on the PutObjectInput (which the code did when AWS_SKIP_CHECKSUM=true) no longer suppresses checksums for multipart uploads — the manager overwrites the cleared value with CRC32 before the S3 client sees it, bypassing the RequestChecksumCalculation = WhenRequired configuration. Fix this by adding smithy-go Initialize middleware via the Uploader's ClientOptions that clears ChecksumAlgorithm from CreateMultipartUploadInput, UploadPartInput, CompleteMultipartUploadInput, and PutObjectInput before the S3 client's checksum middleware processes them. This runs after the manager's initChecksumAlgorithm but before the client computes checksums, effectively suppressing all checksum computation on multipart uploads when AWS_SKIP_CHECKSUM=true. Resolves: cockroachdb#166090 Release note (bug fix): Fixed a bug where the AWS_SKIP_CHECKSUM=true storage option did not fully suppress checksums on multipart uploads to S3-compatible storage. After an internal SDK upgrade, multipart uploads included CRC32 checksums despite the skip flag, which could cause errors with S3-compatible services that do not support CRC checksums. Generated by Claude Code Auto-Solver Co-Authored-By: Claude <noreply@anthropic.com>
|
[autosolve-response] I've addressed the review comments and pushed updates. Changes made: Please review the updated code. |
b7843b0 to
927ff29
Compare
|
/trunk merge |
|
[autosolve-response] I reviewed the comments but no code changes were necessary. Analysis: |
|
You have requested backports to the following End-of-Life (EOL) versions:
While Blathers will still create these backport PRs, please verify that backporting to EOL versions is intentional and appropriate. EOL versions no longer receive maintenance updates according to our support policy. The backport PRs will also include this warning. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
Based on the specified backports for this PR, I applied new labels to the following linked issue(s). Please adjust the labels as needed to match the branches actually affected by the issue(s), including adding any known older branches. Issue #166090: branch-release-26.1. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
After the aws-sdk-go-v2 s3 manager upgrade to v1.17.65, the manager's
internal initChecksumAlgorithm() unconditionally defaults ChecksumAlgorithm
to CRC32 when the field is empty. This means setting ChecksumAlgorithm = ""
on the PutObjectInput (which the code did when AWS_SKIP_CHECKSUM=true) no
longer suppresses checksums for multipart uploads — the manager overwrites
the cleared value with CRC32 before the S3 client sees it, bypassing
the RequestChecksumCalculation = WhenRequired configuration.
Fix this by adding smithy-go Initialize middleware via the Uploader's
ClientOptions that clears ChecksumAlgorithm from CreateMultipartUploadInput,
UploadPartInput, CompleteMultipartUploadInput, and PutObjectInput before
the S3 client's checksum middleware processes them. This runs after the
manager's initChecksumAlgorithm but before the client computes checksums,
effectively suppressing all checksum computation on multipart uploads when
AWS_SKIP_CHECKSUM=true.
Resolves: #166090
Release note (bug fix): Fixed a bug where the AWS_SKIP_CHECKSUM=true
storage option did not fully suppress checksums on multipart uploads
to S3-compatible storage. After an internal SDK upgrade, multipart
uploads included CRC32 checksums despite the skip flag, which could
cause errors with S3-compatible services that do not support CRC
checksums.
Generated by Claude Code Auto-Solver
Co-Authored-By: Claude noreply@anthropic.com
This PR was auto-generated by issue-autosolve using Claude Code.
Please review carefully before approving.