ecr-credential-provider: use custom AWS_PROFILE

[johanvandeweerd]

Image I'm using:

apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "7452c37e",
    "pretty_name": "Bottlerocket OS 1.18.0 (aws-k8s-1.28)",
    "variant_id": "aws-k8s-1.28",
    "version_id": "1.18.0"
  }
}

What I expected to happen:

Setting AWS_PROFILE environment variable for ecr-credential-provider configures the AWS profile used by the ecr-crendentials-provider.

What actually happened:

The AWS_PROFILE environment variable is set twice: once with the configured value, once with value default.

How to reproduce the problem:

Add an AWS profile that is used by ecr-credential-provider:

[settings.aws]
config = "W3Byb2ZpbGUgZWNyXQpyb2xlX2FybiA9IGFybjphd3M6aWFtOjo8QUNDT1VOVF9JRD46cm9sZS9lY3ItcmVhZC1vbmx5CmNyZWRlbnRpYWxfc291cmNlID0gRWMySW5zdGFuY2VNZXRhZGF0YQo="

The config value base64-decoded equals to

[profile ecr]
role_arn = arn:aws:iam::<ACCOUNT_ID>:role/ecr-read-only
credential_source = Ec2InstanceMetadata

Set the AWS_PROFILE using apiclient:

[ssm-user@control]$ enter-admin-container
[root@admin]# apiclient set --json '{"settings":{"kubernetes":{"credential-providers":{"ecr-credential-provider":{"enabled":true,"image-patterns":["*.dkr.ecr.*.amaonaws.com"],"environment":{"AWS_PROFILE":"ecr"}}}}}}'
[root@admin]# cat /.bottlerocket/rootfs/etc/kubernetes/kubelet/credential-provider-config.yaml
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
  - name: ecr-credential-provider
    matchImages:
      - "*.dkr.ecr.*.amaonaws.com"
    defaultCacheDuration: "12h"
    apiVersion: credentialprovider.kubelet.k8s.io/v1
    env:
      - name: AWS_PROFILE
        value: 'ecr'
      - name: HOME
        value: '/root'
      - name: AWS_PROFILE
        value: 'default'

Also tried the following userData with Karpenter but that doesn't seem to be picked up. Will file an issue with Karpenter.

[settings.aws]
config = "W3Byb2ZpbGUgZWNyXQpyb2xlX2FybiA9IGFybjphd3M6aWFtOjo8QUNDT1VOVF9JRD46cm9sZS9lY3ItcmVhZC1vbmx5CmNyZWRlbnRpYWxfc291cmNlID0gRWMySW5zdGFuY2VNZXRhZGF0YQo="

[settings.kubernetes.credential-providers.ecr-credential-provider]
enabled = true
image-patterns = ["*.dkr.ecr.*.amazonaws.com"]

[settings.kubernetes.credential-providers.ecr-credential-provider.environment]
AWS_PROFILE = "ecr"

Additional information:

We have an EKS cluster in AWS account A and want it to pull images from the ECR registry of AWS account B. We try to configure the ecr-credential-provider to use an AWS profile that assumes a role of account B that has readonly access to the ECR registry of account B.

I've seen we can also configure settings.aws.profile and set that (in this case) to ecr. This seems to work but does that mean everything that needs AWS services are using the ecr profile? We want to restrict ONLY the ecr-credential-provider to the ecr role/profile.

[settings.aws]
config = "W3Byb2ZpbGUgZWNyXQpyb2xlX2FybiA9IGFybjphd3M6aWFtOjo8QUNDT1VOVF9JRD46cm9sZS9lY3ItcmVhZC1vbmx5CmNyZWRlbnRpYWxfc291cmNlID0gRWMySW5zdGFuY2VNZXRhZGF0YQo="
profile = ecr

[vyaghras]

@johanvandeweerd Thanks for raising the issue. We will look in to this and come back to you.

[arnaldo2792]

Hi @johanvandeweerd, sorry for the late response.

This seems to work but does that mean everything that needs AWS services are using the ecr profile?

Currently, the only place in Bottlerocket where settings.aws.profile is used is on the configuration for the ECR credentials provider. So any changes you make to this configuration will only affect this component, which is good for your use case 🎉 . However, since this API lives under aws in the API, we might try to use it in the future to set the profile for other services. For the time being, that's not the case so you should be safe, and you can configure the ECR credentials provider as you described above. I'll bring this to the team to discuss how we could better support use cases like yours.

[johanvandeweerd]

Thanks for the response @arnaldo2792.

Will you keep this issue open to track the outcome of any team discussions?

My biggest concern is to depend on this feature and it breaking down the road when other services also start to use that AWS profile because we have a very specific and limited role that we want to use for the credentials provider.

[arnaldo2792]

Yeah, I'll keep this issue open for discussion since I think your use case is something that others might run into later. I'll post an update later on 👍 .