Allow assuming multiple web identity roles

[hasheddan]

What problem are you facing?

A number of provider-aws users make use of the IRSA authentication flow to allow the provider to authenticate to AWS. This is a useful pattern, but restricts the provider Pod to only assuming one role because the EKS Pod Identity Webhook injects this role based on the Pod ServiceAccount, and hardcodes the environment variables that point to the volume mount, and we hardcode accessing them.

How could Crossplane help solve your problem?

Users can choose to manually specify environment variables with multiple ARNs and specific mount paths. In the linked example, two different containers are making use of the same SA token mount, but using different ARNs. Crossplane users may define additional environment variables on their provider containers using the ControllerConfig. This means that if we added a field to the provider-aws ProviderConfig that allowed for the InjectedIdentity source to be customized with something like overrideARNEnvVar, users could have a single provider Pod assume multiple roles using a single SA.

apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: example
spec:
  credentials:
    source: InjectedIdentity
    overrideARNEnvVar: AWS_ROLE_ARN_2

[lewismarshall]

As an alternative to creating multiple web identities to associate with the controller process POD, a new "assume role" capability could be added, see #606.

This would present the advantage that new roles can be created as new resources are enabled without any re-deploy or update to the provider controller pod.

E.g. (repeated here for clarity) a provider config with specific roles:

apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: example
spec:
  credentials:
    source: InjectedIdentity
    assumeRoleARN: "arn:aws:iam::xyz:role/my-specific-resource" 

[benagricola]

Revisiting this so that my musings in Slack are not lost 😃

I think a combination of role assumption using sts:AssumeRoleWithWebIdentity and sts:AssumeRole would cover all bases:

First-off, you need a credential of some sort to be able to assume a role - with InjectedIdentity this is the Web Identity token (contents of AWS_WEB_IDENTITY_TOKEN_FILE) injected by the EKS admission controller with an IRSA configured role.

The role identifier in the ASSUME_ROLE_ARN environment variable is the role that the service account is assigned to, HOWEVER the token is valid on the OIDC provider, so can actually be used to assume any role via web identity where the current OIDC provider is trusted.

We currently call external.LoadDefaultAWSConfig from github.com/aws/aws-sdk-go-v2/aws/external, which does this role assumption via WebIdentity automatically, but we can only override this on a provider-wide basis by setting the AWS_ROLE_ARN environment variable using a ControllerConfig.

1. In the absence of any more specific role configuration, we should continue to simply call external.LoadDefaultAWSConfig(), as this will automatically grant us access to the 'current' AWS account.

2. To solve Allow assuming multiple web identity roles #597, We should add a new ProviderConfig field to allow assumption of a single role via sts:AssumeRoleWithWebIdentity, e.g. assumeWebIdentityRole. When specified, this would not run external.LoadDefaultAWSConfig, and would add a stscreds.NewWebIdentityRoleProvider as the first provider in a new credentials chain, allowing us to override the role ARN to be assumed using the injected OIDC token (this is the equivalent of overrideARNEnvVar above but without the extra indirection towards another env var).

3. To solve Enable specific role assumption in ProviderConfig #606, we should add a second new field on ProviderConfig, e.g. assumeRoles, to allow further assumption of roles via sts:AssumeRole. AssumeRole can be chained multiple times (I do not know the limit to this) as it uses AWS access / secret / session tokens, so this could be represented as a string array if we wanted to allow role chaining. This would add a new stscreds.NewAssumeRoleProvider for each entry to the returned credentials chain, using either the default role assumed by LoadDefaultAWSConfig, or the previously set WebIdentity role as the initial credential source.

Authorization flow would be as follows:

With assumeWebIdentityRole and assumeRoles unset:

• The service account role is assumed by LoadDefaultAWSConfig. We're targeting whatever account that the role assigned to the service account was in (that could be a remote account, but is provider-wide).
• We do not override the credentials chain at all.

With assumeWebIdentityRole set and assumeRoles unset:

• The role ARN in assumeWebIdentityRole is assumed using the injected OIDC token from AWS_WEB_IDENTITY_TOKEN_FILE - the role name in AWS_ROLE_ARN is ignored.
• At this point, we're targeting whatever account the role in assumeWebIdentityRole was in. This requires the target role to be trusting the current OIDC provider.
• This allows us to assume a role via web identity in any number of AWS accounts (each a separate ProviderConfig).

With assumeWebIdentityRole unset and assumeRoles set:

• The service account role is assumed by LoadDefaultAWSConfig.
• Each role in assumeRoles is assumed using the credentials returned by the 'previous' role.
• For the first role in the assumeRoles list, this is assumed by credentials from LoadDefaultAWSConfig (i.e. the Service account).
• This allows us to chain-assume a role in any number of AWS accounts where our service account is trusted (this requires less setup than the assumeWebIdentityRole as it does not need the target roles to be aware of the OIDC provider - each target role only needs to trust the account of the previous role.

With both assumeWebIdentityRole and assumeRoles set:

• The role ARN in assumeWebIdentityRole is assumed as above - default config is not used.
• Each role in assumeRoles is assumed using the credentials returned by the 'previous' role.
• For the first role in the assumeRoles list, this is assumed by credentials from the assumeWebIdentityRole.
• This allows the web identity role to be used as an intermediate before further role assumption.
• I can't think of a good reason for wanting to do this unless you somehow have an intermediate account which trusts the 'service acount' via WebIdentity and not as an account principal.

Thoughts on this? I don't think the implementation is remotely as complex as the explanation here, but it would be good if we could simplify this further (e.g. rule out using both assumeWebIdentityRole and assumeRoles or maybe finding a way to express them as a single field).

Happy to work on this as I've been poking around this stuff already but I'd like to confirm my thinking around the approach is sound before I go further.

[jessesanford]

I am interested in @benagricola 's second scenario above:

With assumeWebIdentityRole set and assumeRoles unset:
The role ARN in assumeWebIdentityRole is assumed using the injected OIDC token from AWS_WEB_IDENTITY_TOKEN_FILE - the role name in AWS_ROLE_ARN is ignored.
At this point, we're targeting whatever account the role in assumeWebIdentityRole was in. This requires the target role to be trusting the current OIDC provider.
This allows us to assume a role via web identity in any number of AWS accounts (each a separate ProviderConfig).

I have a situation where we can allow other account's IDPs to trust our K8s cluster's OIDC provider but we cannot use sts chaining on roles due to a explicit boundary limitation.

You can see that the suggestion given in the aws pod identity webhook comments thread does work inside my provider pod once the IDP trust to to the oidc is setup:

bash-4.2# aws sts get-caller-identity
{
    "UserId": "FEADBEEF:botocore-session-1643293101",
    "Account": "555555555555",
    "Arn": "arn:aws:sts::555555555555:assumed-role/Superman/botocore-session-1643293101"
}
bash-4.2# export AWS_ROLE_ARN=arn:aws:iam::666666666666:role/Deployer
bash-4.2# aws sts get-caller-identity
{
    "UserId": "DEADBEEF:botocore-session-1643293197",
    "Account": "666666666666",
    "Arn": "arn:aws:sts::666666666666:assumed-role/BizzaroSuperman/botocore-session-1643293197"
}
bash-4.2# export AWS_ROLE_ARN=arn:aws:iam::777777777777:role/Deployer
bash-4.2# aws sts get-caller-identity
{
    "UserId": "18BADBEEF:botocore-session-1643293245",
    "Account": "77777777777",
    "Arn": "arn:aws:sts::777777777777:assumed-role/BizzaroJerry/botocore-session-1643293245"
}

What is the current workaround? To have multiple aws providers?
How can we achieve connecting to multiple client aws accounts with a single instance of Crossplane?

[hasheddan]

@jessesanford does the new functionality @haarchri introduced in #912 that has now been included in v0.23.0 (https://doc.crds.dev/github.com/crossplane/provider-aws/aws.crossplane.io/ProviderConfig/v1beta1@v0.23.0#spec-assumeRoleARN) fit your use case?

[haarchri]

check: https://github.com/crossplane/provider-aws/blob/master/AUTHENTICATION.md#how-can-assumerolearn-be-used-with-provider-aws-

[jessesanford]

@jessesanford does the new functionality @haarchri introduced in #912 that has now been included in v0.23.0 (https://doc.crds.dev/github.com/crossplane/provider-aws/aws.crossplane.io/ProviderConfig/v1beta1@v0.23.0#spec-assumeRoleARN) fit your use case?

Doesn't this imply an STS Assumption as when the controller connects to the Amazon api in the client account before it starts managing resources? From what I see in the code this is an AssumeRole call and not an AssumeRoleWithWebIdentity which would be exchanging the original jwt token for the role. While we can get our client accounts setup with an IDP that trusts the k8s server hosting crossplane, we cannot use cross account role trusts for sts chaining.

[cdenneen]

@jessesanford I just ran into this issue where I needed to add on AccountA role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::ACCOUNTB:role/provider-aws"
        }
    ]
}

and on AccountB role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::ACCOUNTB:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/XXXX"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/XXXX:aud": "sts.amazonaws.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNTA:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

first part of this second statement should be enough if switched from sts:AssumeRole to sts:AssumeRoleWithWebIdentity like the default eks.amazonaws.com/role-arn annotation does.

[cdenneen]

if I fully comprehend the limitation then most likely what would need to happen is, in these cases where adding all these role trusts is a bit unwieldy the better solution would be to create multiple Provider which all have their own ControllerConfig... this creates multiple Deployments and ServiceAccounts and then the ProviderConfig instead of using assumeRoleARN would need to pick which Provider it wanted to use (i.e. which pod it wanted to use for Auth)... then since each of these serviceAccounts has it's own eks.amazonaws.com/role-arn annotation they would all be using the IRSA OIDC trust. Does that sound about right?

This would just require the ProviderConfig have an option to choose which Provider to leverage.

(This is all only necessary if @benagricola 's solution couldn't be implemented which would leverage the single Deployment).

[jessesanford]

@nabuskey and I have been working on similar solutions here: #1258 and here: nabuskey/provider-aws@master...nabuskey:feature/oidc-assume-role if anyone wants to take a look.