BSL.spec.config["credentialsFile"] overrides AWS Web Identity Token credentials from pod environment

[RobKenis]

Describe the problem/challenge you have
We have Velero deployed on AWS EKS with IRSA. This allows the pod to assume a role with permissions on bucket A. We have another bucket (B) in another AWS account to which we want to grant permissions using an IAM User. We have the credentials configured on the BackupStorageLocation like so:

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: cross-account-aws-bucket
  namespace: velero
spec:
  provider: velero.io/aws
  objectStorage:
    bucket: cross-account-aws-bucket
  credential:
    key: cloud
    name: aws-credentials
  config:
    region: eu-central-1

The issue is that Velero authenticates to the bucket using the IAM Role which is assumed through IRSA and does not take the static credentials into account. This could be because the default credentials chain gives priority to web identity credentials over the shared credentials file: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html

Describe the solution you'd like
When explicit credentials are defined on a BackupStorageLocation, these always take first priority.

Environment:

• Velero version (use velero version): 5.2.0
• Kubernetes version (use kubectl version): 1.28.4
• Kubernetes installer & version: EKS
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): N/A

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "The project would be better with this feature added"
• 👎 for "This feature will not enhance the project in a meaningful way"

[RobKenis]

Credentials chain for Go is unclear to me: https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/
AWS_WEB_IDENTITY_TOKEN_FILE is set as higher priority than Shared configuration files, so I'm assuming it's the same issue.

[reasonerjt]

@RobKenis
Thanks for reporting this issue, I think this is a valid use case and I'll look into this.
A shortcut is probably to unset the environment variables when the credential file is provided via BSL.

[RobKenis]

@reasonerjt We cannot unset the environment variables because we still need those for authenticating to S3 for our main backup bucket. I've attached a PR on how we may potentially work around the issue. If we support a custom credentials provider chain in the aws plugin, we can set the custom provider (like a Kubernetes secret) in the config of the BSL and then it should override the default credentials provider chain. Could you provide some feedback on this?

[kaovilai]

I would retitle this issue to BSL.spec.config["credentialsFile"] overrides AWS Web Identity Token credentials from pod environment if the solution to use credentialsFile works for you.

[danfengliu]

nightly log