Search before asking
Paimon version
master @ b85955c (2.1-SNAPSHOT); also present in release-2.0
Compute Engine
Flink and Spark (format table writes)
Minimal reproduce step
- Set
s3.encryption.algorithm=SSE-C and s3.encryption.key=<base64 256-bit key>.
- INSERT into a format table on that bucket.
S3MultiPartUpload.startMultiPartUpload() initiates the upload through s3accessHelper, but uploadPart() (S3MultiPartUpload.java line 88) hand-assembles UploadPartRequest.builder()...build(), bypassing the S3A request factory. AWS requires identical encryption information on every part of an upload initiated with SSE-C, so S3 rejects the part upload (HTTP 400).
MultiPartUploadTwoPhaseOutputStream.closeForCommit() calls uploadPartUtil() unconditionally, so every non-empty format table file is affected regardless of size. SSE-S3, SSE-KMS and DSSE-KMS are unaffected.
What doesn't meet your expectations?
Writing a format table on an SSE-C bucket should succeed; it fails instead.
Anything else?
Regression from #7187 (cca171f43), which replaced s3accessHelper.newUploadPartRequest(...) with a hand-built request while keeping the helper for initiate/complete/abort. hadoop-aws itself always builds part requests through the factory (S3ABlockOutputStream.java:1033), the path normal Paimon table writes take, so those are unaffected.
Are you willing to submit a PR?
Search before asking
Paimon version
master @ b85955c (2.1-SNAPSHOT); also present in release-2.0
Compute Engine
Flink and Spark (format table writes)
Minimal reproduce step
s3.encryption.algorithm=SSE-Cands3.encryption.key=<base64 256-bit key>.S3MultiPartUpload.startMultiPartUpload()initiates the upload throughs3accessHelper, butuploadPart()(S3MultiPartUpload.javaline 88) hand-assemblesUploadPartRequest.builder()...build(), bypassing the S3A request factory. AWS requires identical encryption information on every part of an upload initiated with SSE-C, so S3 rejects the part upload (HTTP 400).MultiPartUploadTwoPhaseOutputStream.closeForCommit()callsuploadPartUtil()unconditionally, so every non-empty format table file is affected regardless of size. SSE-S3, SSE-KMS and DSSE-KMS are unaffected.What doesn't meet your expectations?
Writing a format table on an SSE-C bucket should succeed; it fails instead.
Anything else?
Regression from #7187 (
cca171f43), which replaceds3accessHelper.newUploadPartRequest(...)with a hand-built request while keeping the helper for initiate/complete/abort. hadoop-aws itself always builds part requests through the factory (S3ABlockOutputStream.java:1033), the path normal Paimon table writes take, so those are unaffected.Are you willing to submit a PR?