Vault not using role provided by IAM Roles for Service Accounts (IRSA)

[kirkdave]

Describe the bug
When running Vault on EKS (deployed via Helm chart) it is not using the IAM role annotated on the service account to get permissions for AWS API calls. It uses the IAM role assigned to the worker node

Error parsing Seal configuration: error fetching AWS KMS wrapping key information: AccessDeniedException: User: arn:aws:sts::XXXXXXXXXXX:assumed-role/{worker node role}/{instance id} is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:eu-west-2:XXXXXXXXXXX:key/{kms key id}
	status code: 400, request id: ad518429-74c8-4fdc-b8a2-65916ada2666

To Reproduce
Steps to reproduce the behavior:

1. Create a KMS Key in AWS
2. Create a an EKS Cluster, with OIDC provider (https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html)
3. Create an IAM role with the required trust relationship for the OIDC provider and service account and permissions on the KMS key created in 1 (https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html)
4. Deploy Vault using the official Helm chat with following vaules:

server:
  affinity: |
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: {{ template "vault.name" . }}
                release: "{{ .Release.Name }}"
                component: server
            topologyKey: kubernetes.io/hostname

  serviceAccount:
    create: true
    annotations: |
      eks.amazonaws.com/role-arn: "${iam_role_arn}"

  ingress:
    enabled: false

  service:
    enabled: true

  ha:
    enabled: true
    replicas: 1

    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      storage "consul" {
        path = "vault"
        address = "HOST_IP:8500"
      }

      seal "awskms" {
          region      = "${region}"
          kms_key_id = "${kms_key_id}"
          endpoint   = "https://kms.${region}.amazonaws.com"
      }

      log_level = "Debug"
  extraEnvironmentVars:
    VAULT_LOG_LEVEL: debug
    AWS_ROLE_SESSION_NAME: vault

injector:
  enabled: true
ui:
  enabled: true

Expected behavior
Vault should use the role annotated on the service account to authenticate with AWS and successfully describe the KMS key

Environment:

• Vault Server Version: 1.5.4 and 1.6.0
• Server Operating System/Architecture: Official Vault Docker image

Additional context
To validate the IAM role can be successfully assumed using IRSA I have run the following command, which was successful

kubectl run --serviceaccount=vault --rm -i --tty --attach amazonlinux --image=amazonlinux -- /bin/bash -c "yum update -y && yum install awscli -y && aws sts get-caller-identity && aws sts assume-role-with-web-identity --role-session-name test --role-arn {iam_role_arn} --web-identity-token file://var/run/secrets/eks.amazonaws.com/serviceaccount/token"

Which returns

{
    "Account": "726434132247", 
    "UserId": "AROA2SIWZGEL2NKM3N5DT:botocore-session-1606390510", 
    "Arn": "arn:aws:sts::XXXXXXXXXX:assumed-role/{role name}/botocore-session-1606390510"
}
{
    "AssumedRoleUser": {
        "AssumedRoleId": "AROA2SIWZGEL2NKM3N5DT:test", 
        "Arn": "arn:aws:sts::XXXXXXXXXX:assumed-role/{role name}/test"
    }, 
    "Audience": "sts.amazonaws.com", 
    "Provider": "arn:aws:iam::XXXXXXXXXX:oidc-provider/oidc.eks.eu-west-2.amazonaws.com/id/9C041BD4B422683092XXXXXXXXXX", 
    "SubjectFromWebIdentityToken": "system:serviceaccount:default:vault", 
    "Credentials": {
        "SecretAccessKey": "{valid secret access key}", 
        "SessionToken": "{valid session token}", 
        "Expiration": "2020-11-26T12:35:13Z", 
        "AccessKeyId": "{valid access key}"
    }
}

[kirkdave]

I got this to work by manually adding the AWS_ROLE_ARN environment variable to the deployment. Not sure if this is expected to be manually added, couldn't find anything to say either way

[swayne275]

Thanks for pointing this out (and providing the solution)! It looks like setting that env variable is part of the aws setup, but either way i've labeled it so the docs team can take a look.

[Chili-Man]

@kirkdave what version of EKS are you using? EKS should automatically inject the AWS_ROLE_ARN into the pod as part of the IRSA stuff. I'm running vault on EKS 1.17 and 1.18 with IRSA and I have not ran into this issue (also deployed with Helm).

[worldofgeese]

@kirkdave what version of EKS are you using? EKS should automatically inject the AWS_ROLE_ARN into the pod as part of the IRSA stuff. I'm running vault on EKS 1.17 and 1.18 with IRSA and I have not ran into this issue (also deployed with Helm).

I'm currently running into this issue on an EKS 1.17 cluster. Could I bother you to post your chart override values for Vault?

I've gone into detail at hashicorp/vault-helm#368 (comment)

[worldofgeese]

I've tried now manually setting the AWS_ROLE_ARN. Attempting to init the vault looks like a step backward for me as it attempts to use the root role:

Error initializing: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/init
Code: 400. Errors:

* failed to store keys: failed to encrypt keys for storage: error encrypting data: AccessDeniedException: User: arn:aws:iam::472409228388:root is not authorized to perform: kms:Encrypt on resource: arn:aws:kms:eu-north-1:472409228388:key/260757fd-c5e8-49b4-972d-66fb5a366a1e
        status code: 400, request id: 93afeb85-8540-4f56-9771-918245bcfea4

With the AWS_ROLE_ARN attached automatically, it looks like Vault attempts to use the correct role but without success:

* failed to store keys: failed to encrypt keys for storage: error encrypting dat
a: AccessDeniedException: User: arn:aws:sts::472409228388:assumed-role/vault/161
0984202537365570 is not authorized to perform: kms:Encrypt on resource: arn:aws:
kms:eu-north-1:472409228388:key/260757fd-c5e8-49b4-972d-66fb5a366a1e
        status code: 400, request id: 1abb7c91-0f8c-4175-a745-7f103d07169a

This despite setting the correct policy permissions via Terraform:

data "aws_iam_policy_document" "vault" {
  statement {
    sid    = "VaultKMSUnseal"
    effect = "Allow"

    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:DescribeKey",
    ]

    resources = ["*"]
  }
}

[worldofgeese]

This turned out to be KMS key related. For some reason the original key just didn't want to allow permissions. Created a new key and boom, all is well

[johnsonprabu-a]

I am also facing such issue, i have tried by adding "AWS_ROLE_ARN", to the env. It doesn't consider it.

Also below error is for ServiceAccount.

eal.awskms: error assuming role: roleARN=* tokenPath=/var/run/secrets/eks.amazonaws.com/serviceaccount/token sessionName= err="WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
status code: 403, request id: 9ce88500-9b64-4d6d-b98f-02608eba78c5"

[ksquaredkey]

Ran into a same error message on an older EKS-based Vault that's gone from evolution from KIAM to IRSA. In my case, the issue was that the KMS Key Policy only had the default policy (the one that allows Admin access). This worked in the KIAM environment. I had to explicitly add my IRSA role to the KMS Key Policy to get Vault to auto unseal with KMS when using IRSA. In hindsight, this makes sense. Had to change from:

{
    "Version": "2012-10-17",
    "Id": "key-default-1",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::XXXXXXXXXXXXX:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}

to:

{
    "Version": "2012-10-17",
    "Id": "key-default-1",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::XXXXXXXXXX:root",
                    "arn:aws:iam::XXXXXXXXXX:role/xxxx-irsa-vault-kms-role"
                ]
            },
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}

[nitrogear]

I got it working with Vault v1.14.1
The issue was in SA annotation. If you don't add the annotation "eks.amazonaws.com/role-arn" to the Vault's SA account then EKS won't mount token "/var/run/secrets/eks.amazonaws.com/serviceaccount/token" to vault pod.
Without that Vault ignores role_arn/web_identity_token_file in the configuration file

[adrienbroyere]

Definitively KO in 1.14.0, got it working in Vault v1.14.1 simply by changing the image tag. Every other resources (Service account, annotations, IAM role, KMS policy and so on) were unchanged between the two image tags