[velero-plugin-for-aws:v1.9.1] Backup fails with customer provided key encryption

[robertstrache]

What steps did you take and what happened:
After upgrading velero-plugin-for-aws from 1.8.1 to 1.9.0 (and also 1.9.1) backup creation fails with

Requests specifying Server Side Encryption with Customer provided keys must provide the client calculated MD5 of the secret key

What did you expect to happen:
The backup creation should work correctly as it did previously in 1.8.1 (reverting to 1.8.1 makes it work again)

The following information will help us better understand what's going on:
BackupStorageLocation:

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: default
spec:
  accessMode: ReadWrite
  config:
    customerKeyEncryptionFile: /credentials/customer-key
    region: us-east-1
  default: true
  objectStorage:
    bucket: bucket
  provider: aws

Environment:

• Velero version (use velero version): 1.13.1
• Kubernetes version (use kubectl version): v1.28.6
• Cloud provider or hardware configuration: AWS

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[reasonerjt]

@robertstrache
Could you please provide the debug bundle of velero so I can see the complete log messages?

Could you check the workaround provided in aws-plugin v1.9.2 to skip adding the checksum?

[robertstrache]

When using aws-plugin v1.9.2 and adding

spec:
  config:
    checksumAlgorithm: ""

to the BackupStorageLocation we see the same error.

How do I create the debug bundle? I am using

velero backup create --from-schedule velero-daily -nNAMESPACE

to manually trigger a backup creation. With which version of the aws-plugin should i create the debug bundle?

[reasonerjt]

@robertstrache
Please check velero debug --help, and create the debug bundle for one backup that is failing.

Please gen the debug bundle of aws-plugin v1.9.2

[robertstrache]

I did with version 1.9.2 but I am not sure if I can provide a meaningful debug bundle. What exactly do you need from it?
When I try to trigger the backup the last line reads

Run `velero backup describe velero-daily-20240423065945` or `velero backup logs velero-daily-20240423065945` for more details.

and I assume these are also used during debug bundle creation.

At some point the output of

velero backup describe velero-daily-20240423065945 -nNAMSPACE

says

Phase:  Failed (run `velero backup logs velero-daily-20240423065945` for more information)

and this command ends with

An error occurred: file not found

(which I guess makes sense because it could not communicate with S3)

However e.g. kubectl get backup velero-daily-20240423065945 -o yaml shows in the status

status:
  expiration: "2024-04-30T06:59:45Z"
  failureReason: 'rpc error: code = Unknown desc = error putting object backups/velero-daily-20240423065945/velero-backup.json:
    operation error S3: PutObject, https response error StatusCode: 400, RequestID:
    AYxxxxxxxxxx, HostID: zTxxxxxxxxxx,
    api error InvalidArgument: Requests specifying Server Side Encryption with Customer
    provided keys must provide the client calculated MD5 of the secret key.'
  formatVersion: 1.1.0
  hookStatus: {}
  phase: Failed
  progress:
    itemsBackedUp: 450
    totalItems: 450
  startTimestamp: "2024-04-23T06:59:45Z"
  version: 1

which is also the error we see in our logs. Do you need additional information?
Thank you!

[arteonprifti]

had the same issue when upgrading from v1.8.2 to v1.9.2

time="2024-05-29T08:46:14Z" level=error msg="backup failed" backuprequest=velero/customer-namespaces-20240529084609 controller=backup error="rpc error: code = Unknown desc = error putting object backups/test-20240529084609/velero-backup.json: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 2410e3d9-ffe4-1fff-a0bd-3cecefb1f962, HostID: 1470ea1184f74570b8bfe5f32549d3f0, api error InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must provide the client calculated MD5 of the secret key." logSource="pkg/controller/backup_controller.go:288"

Addding the checksumAlgorithm: "" didn't help either

[EugenMayer]

in my case adding checksumAlgorithm fixes the issue ( even though the flag is not documented in the values.yaml)

configuration:
  backupStorageLocation:
    - name: default
      provider: aws
      bucket: velero
      default: true
      config:
        region: de
        s3ForcePathStyle: "true"
        s3Url: ${minio_uri}
        checksumAlgorithm: ""

[mzimry]

checksumAlgorithm: ""

same here

[edric-le]

Requirement: Using server-side encryption with customer-provided keys
Resource: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html
Repo: https://github.com/vmware-tanzu/velero-plugin-for-aws
Issue: I have investigated the repository velero-plugin-for-aws and realized it wasn't working properly. It receives 32 bytes from customerKeyEncryptionFile and doesn't do anything else before requesting.
Solution: You need to base64-encoded encryption key for SSECustomerKey and base64-encoded 128-bit MD5 digest of the encryption key for SSECustomerKeyMD5.

Coding(./velero-plugin-for-aws/object_store.go):
func (o *ObjectStore) Init(config map[string]string) error {
...
if customerKeyEncryptionFile != "" {
customerKey, err := readCustomerKey(customerKeyEncryptionFile)
if err != nil {
return err
}
sseCustomerKey := base64.StdEncoding.EncodeToString([]byte(customerKey))
o.sseCustomerKey = sseCustomerKey
hash := md5.Sum([]byte(customerKey))
sseCustomerKeyMd5 := base64.StdEncoding.EncodeToString([]byte(hash))
o.sseCustomerKeyMd5 = sseCustomerKeyMd5
}
...
}
func (o *ObjectStore) PutObject(bucket, key string, body io.Reader) error {
...
case o.sseCustomerKey != "":
input.SSECustomerAlgorithm = aws.String("AES256")
input.SSECustomerKey = &o.sseCustomerKey
input.SSECustomerKeyMD5 = &o.sseCustomerKeyMd5
...
}

After adjusting, building a new image, and using