AWS: Change S3FileIO to use SHA1 based checksums #10293
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue
Using S3 Express for Iceberg was previously failing multipart upload integration tests as well as when
s3.checksum-enabled
was set. This was because S3 Express doesn't support using MD5 based checksums.Fix
Changes S3FileIO to use SHA1 based checksums when checksumming is enabled.
Testing
Ran S3FileIO integration tests against both a regular S3 bucket as well as an S3 Express bucket. For regular S3, all tests pass, and for S3 Express, the checksumming errors are now resolved.