Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bulkio: allow user to request S3 server-side encryption #31612

Closed
rolandcrosby opened this issue Oct 18, 2018 · 2 comments · Fixed by #55834
Closed

bulkio: allow user to request S3 server-side encryption #31612

rolandcrosby opened this issue Oct 18, 2018 · 2 comments · Fixed by #55834
Labels
A-disaster-recovery C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Comments

@rolandcrosby
Copy link

S3 offers transparent server-side encryption, which is enabled by setting a combination of x-amz-server-side-encryption headers on the PutObject request. We should allow the user to pass string values for these options, likely as S3 URL parameters. For full support of server-side encryption, we'd need to allow the user to specify values for the following headers (all of which appear to be available in the PutObjectInput type in the S3 Go SDK):

  • x-amz-server-side-encryption
  • x-amz-server-side-encryption-aws-kms-key-id
  • x-amz-server-side-encryption-context
  • x-amz-server-side​-encryption​-customer-algorithm
  • x-amz-server-side​-encryption​-customer-key
  • x-amz-server-side​-encryption​-customer-key-MD5
@petermattis petermattis added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Oct 19, 2018
@rolandcrosby
Copy link
Author

S3 Default Encryption mostly makes this irrelevant. We will reopen if there is significant customer demand to pass custom options for this in the future.

@adityamaru
Copy link
Contributor

Fleshing out some details here:
AWS supports two forms of SSE - AES256 and KMS

KMS requires:
x-amz-server-side-encryption set to aws:kms
x-amz-server-side-encryption-aws-kms-key-id set to the ID referencing the KMS CMK

AES 256 requires:
x-amz-server-side-encryption set to AES256 if the customer is okay with amazon managed encryption keys.

OR if the customer would like to specify a custom key and MD5 then:
x-amz-server-side​-encryption​-customer-algorithm must be set to AES256 along with
x-amz-server-side​-encryption​-customer-key, x-amz-server-side​-encryption​-customer-key-MD5.

No other customer algorithm is supported by AWS according to https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeysSSEUsingRESTAPI.html. The question boils down to what subset of these options would we like to support when writing to external storage? The decision on whether to use URI params or stringy WITH options can be based on the degree of flexibility we need.

craig bot pushed a commit that referenced this issue Oct 29, 2020
55834: storage: support AWS KMS server side encryption r=pbardea,dt a=adityamaru

S3 URIs now support two additional params to provide server side
encryption:

AWS_SERVER_ENC_MODE: can be set to either AES256 or aws:kms

AWS_SERVER_KMS_ID: if the above is set to aws:kms then this is the ID of
the CMK to be used

Closes: #31612

Release note (api change): `AWS_SERVER_ENC_MODE` and `AWS_SERVER_KMS_ID`
can now be specified as a param in all s3 URIs which write to the store.
(egs: Changefeed, BACKUP)

Co-authored-by: Aditya Maru <adityamaru@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-disaster-recovery C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants