external-secrets not assuming web identity role

[vitorfhc]

Hello,

I decided to install external-secrets using its kubernetes version.

For authenticating I chose the IAM roles for service accounts which provides the following environment variables:

AWS_ROLE_ARN=arn:aws:iam::<account-id>:role/<role-to-be-assumed>
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

But after trying to create an external secret I got the following error:

User: arn:aws:sts::<account-id>:assumed-role/<node role> is not authorized to perform: secretsmanager:GetSecretValue on resource: <secret-name> because no identity-based policy allows the secretsmanager:GetSecretValue action

This means external secrets didn't try to run assumeRoleWithWebIdentity using the given token to assume AWS_ROLE_ARN which has the right permissions.

How could I assume the right role? Is there any missing configuration for me?

[gusfcarvalho]

Hey @vitorfhc ! I think you got your issue covered, but just wanted to tell that both projects are implementations for kubernetes. kubernetes-external-secrets is on maintenance mode (see external-secrets/kubernetes-external-secrets#864), so, if possible, try using external-secrets. Most of the features are compatible between the two projects.

[vitorfhc]

@gusfcarvalho thanks for that! Just changed my helm charts to use external-secrets. Had no idea it was going through maintenance.

[meysam81]

Solution

For anyone else getting to this point from Google, I had a bare-metal Kubernetes cluster (it's actually AKS, but I consider it bare-metal in the case of AWS EKS :) ).
The External-Secrets operator (v0.9.16) did not work as I expected, so I dug through the source code.

Objective

I wanted to set up the AWS IAM OIDC provider using the Azure AKS issuer URL.

When the OpenID Configuration trust was set up, my aim was to allow the external-secrets ServiceAccount to be able to assume a role in AWS using Web Identity.

I went through some back and forth. But eventually, and if your environment setup is anything like me, I ended up with the following:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-parameter-store
spec:
  provider:
    aws:
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
      region: eu-central-1
      service: ParameterStore

And I did no change to the helm deployment of the External Secrets.

However, I had to manually patch and annotate the external-secrets ServiceAccount to be in a format that is expected from an SA in an AWS AKS Kubernetes cluster.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/audience: sts.amazonaws.com  # <- add this
    eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXX:role/external-secrets  # <- and this, manually!
    meta.helm.sh/release-name: external-secrets
    meta.helm.sh/release-namespace: external-secrets
  labels:
    app.kubernetes.io/instance: external-secrets
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: external-secrets
    app.kubernetes.io/version: v0.9.16
    helm.sh/chart: external-secrets-0.9.16
  name: external-secrets
  namespace: external-secrets

Problem No. 1

This is a bit odd in my humble opinion. The reason is that this will restrict my External Secrets deployment to a single IAM Role. Imagine needing to have multiple SecretStores, each with a different IAM role. This will restrict the controller down to just one!

Another odd thing here is that you can't specify your audience in your SecretStore. This is, IMO, a bug. And the source is here:

external-secrets/pkg/provider/aws/auth/auth.go

Lines 261 to 264 in 3d96be0

	audiences := []string{tokenAud}
	if len(auth.JWTAuth.ServiceAccountRef.Audiences) > 0 {
	audiences = append(audiences, auth.JWTAuth.ServiceAccountRef.Audiences...)
	}

This means that if I specify audience in my SecretStore, I will have two audiences in the final token. And AWS does not like that and will complain about having multiple audiences in the token.

could not validate provider: WebIdentityErr: failed to retrieve credentials
caused by: InvalidIdentityToken: Token audience contains more than one audience while authorized party is not present

Problem No. 2

The stragest thing happens when you add role to this as follows:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-parameter-store
spec:
  provider:
    aws:
      auth:
        jwt:
          serviceAccountRef:
            audiences:
            - sts.amazonaws.com
            name: external-secrets
            namespace: external-secrets
      region: eu-central-1
      role: arn:aws:iam::XXXXXXXXXXXX:role/some-poor-role  # <- this should be normal, right? right!?
      service: ParameterStore

This gets really hectic really fast.

could not validate provider: AccessDenied: User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/external-secrets/external-secrets-provider-aws is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::XXXXXXXXXXXX:role/external-secrets

And the source code for this behavior is here:

external-secrets/pkg/provider/aws/auth/auth.go

Lines 310 to 311 in 3d96be0

	return stscreds.NewWebIdentityRoleProviderWithOptions(
	sts.New(sess), roleArn, "external-secrets-provider-aws", tokenFetcher), nil

For some reason, unknown to me, External Secrets tries to assume a role before actually assuming my own role. It even falsely tries sts:AssumeRole and not the actual sts:AssumeRoleWithWebIdentity I wanted it to!

Problem No. 3

The current hacky way of annotating the external-secrets Service Accounts means that the Deployment/external-secrets will only get to have one assumable role attached to it. Again, what if we want to allow assuming multiple role from more than one AWS account. This applies to both ClusterSecretStore and SecretStore nonetheless. 🤷

If you were thinking of creating multiple Service Accounts, though luck! Kubernetes allows only a single Service Account per pod.

Expected Behavior

At the very least, I would like to be able to have multiple SecretStore, each being able to use the same SecretAccount token, even if there's a need for manually mounting the projected token.

This, in essence, means that I SHOULD be able to do this (even if not running inside AWS EKS cause it's a K8s cluster for heaven's sake):

---
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-parameter-store-ABC
spec:
  provider:
    aws:
      auth:
        jwt:
          serviceAccountRef:
            audiences:
            - sts.amazonaws.com  # <- this dude
            name: external-secrets
            namespace: external-secrets
      region: eu-central-1
      role: arn:aws:iam::XXXXXXXXXXXX:role/ABC  # <- and this one
      service: ParameterStore
---
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-parameter-store-XYZ
spec:
  provider:
    aws:
      auth:
        jwt:
          serviceAccountRef:
            audiences:
            - sts.amazonaws.com  # <- and this
            name: external-secrets
            namespace: external-secrets
      region: eu-central-1
      role: arn:aws:iam::XXXXXXXXXXXX:role/XYZ  # and also this, SHOULD all be possible!
      service: ParameterStore

Blog Post

I have nothing else to add here to answer the title of this issue. But, at the same time, it would be a waste if I didn't do a shameless self-promotion to the same topic I'll be publicly writing about in my blog and readily available in a few days (by the time you read this, it's already available 😁 ).

https://developer-friendly.blog/2024/04/29/external-secrets-operator-fetching-aws-ssm-parameters-into-azure-aks/