(cli): (cdk fails with "Profile <profile name> did not include credential process)

[kirnberger1980]

Describe the bug

We want to use the CDK CLI with Gitlab Runner, which is running on EKS in Account A. cdk deploy is executed for Account B (Cross-Account Deployment).

cdk deploy --profile AccountB

CLI Command fails with:

Unable to determine the default AWS account (ProcessCredentialsProviderFailure): Profile <AccountB> did not include credential process
....
Could not assume arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1, proceeding anyway.

We are using IRSA (IAM Role for Service Accounts), further called Role A, for the Gitlab Runner in Account A.
In Account B there is an IAM Role (Role B), which can be assumed by Role A. RoleB can assume all cdk-provisioned IAM Roles in that Account (created by cdk bootstrap).

Profile file (stored in /configmaps/awsconfig):

    [default]
    region = eu-central-1
    role_arn = arn:aws:iam::AccountA:role/RoleA
    web_identity_token_file=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
    output = json


    [profile AccountB]
    role_arn = arn:aws:iam::AccountB:role/RoleB
    source_profile = default
    output = json
    region = eu-central-1

We have set environment variables AWS_PROFILE and AWS_CONFIG_FILE:

AWS_PROFILE=AccountB
AWS_CONFIG_FILE=/configmaps/awsconfig

AWS CLI is working fine with above profile file and the environment variables:

aws sts get-caller-identity
{
    "UserId": "AROA2SFT2XXXXXXXXX:botocore-session-168600000",
    "Account": "00000000000",
    "Arn": "arn:aws:sts::00000000000:assumed-role/RoleB/botocore-session-168600000"
}

Assuming the cdk-lookup Role from RoleB with AWS CLI works fine.

aws sts assume-role --role-arn arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1 --role-session-name test123
{
    "Credentials": {
        "AccessKeyId": "<somekey>",
        "SecretAccessKey": "<somesecret>",
        "SessionToken": "<nice jwt>",
        "Expiration": "2023-06-06T15:29:36+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "<some otherkey:test123",
        "Arn": "arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1"
    }
}

By the way: cdk also seems not to respect AWS_PROFILE environment variable as it is using the default role (it could be, that the session token by IRSA is used), when NOT using --profile.

Expected Behavior

We expect that AWS CDK CLI is acting identically to AWS CLI when using AWS profiles for Cross-Account Deployments. Environment Variable AWS_PROFILE should be respected.

Current Behavior

The CDK CLI fails that the cdk-Lookup-Role cannot be assumed, because "Profile did not include credential process". When you don't specify --profile, then default-Profile (or session token created by OIDC Provider of IAM) is used.

Reproduction Steps

You need to run a pod in EKS:

• configure IRSA (RoleA) for the pod in the same account (Account A)
• configure a RoleB in Account B, which can be assumed by RoleA
• give permissions to RoleB to assume cdk-provisioned roles
• configure an AWS_CONFIG_FILE like described above.
• install CDK, NODE etc on the pod
• Configure env variables AWS_PROFILE and AWS_CONFIG_FILE
• Run cdk deploy

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.78.0

Framework Version

2.78.0

Node.js Version

v18.16.0

OS

node:18-alpine

Language

Typescript

Language Version

3.9.7

Other information

No response

[pahud]

Thanks for reporting.

1. Can you run this in the pod from Account A?

aws --profile AccountB sts get-caller-identity

2. Can you share a minimal CDK code that provisions this environment so I can try reproduce it in my account?

[kirnberger1980]

Thanks for the quick reply.

1. Here is the result of the aws cli response:

{
    "UserId": "AXXXXXXXXXX:botocore-session-1686146634",
    "Account": "<AccountB>",
    "Arn": "arn:aws:sts::<AccountB>:assumed-role/RoleB/botocore-session-1686146634"
}

This is, what we expected, as setting env variable AWS_PROFILE or using --profile should return the identical response.

2. Here is some sample code tested, where we could reproduce the issue:

bin-File:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { CdkStack } from '../lib/cdk-stack';

const app = new cdk.App();
new CdkStack(app, 'CdkStack', {
  env: { account: "<AccountB>",region: "eu-central-1" }
});

lib-File:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class CdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

  const vpc = cdk.aws_ec2.Vpc.fromLookup(this, "MyVPC", {
    vpcName: "MyVpcName"
  });

  new cdk.aws_ec2.SecurityGroup(this, "MyNewSecurityGroup", {
    vpc,
    description: "This will secure my resources"
  });
  }
}

Kind Regards,
Marcell

[kirnberger1980]

Any news on this issue?

[github-actions]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

[jumpinjan]

I am facing a similar issue. An update would be greatly appreciated.

[elixxx]

Whats going on with this issue? We are facing the same issue but sounds like a common workflow!

[megakoresh]

Is this a recent thing? I just now saw this. CDK worked totally fine then suddenly it's now failing with this error and in the verbose log I see profile xxx did not include credential process. I have the profile name exported in the AWS_PROFILE variable. I reset the terminal and ran aws sso login again, but nothing helped. Removal of cdk.out folda also didn't help. It's as if it broke as a result of some change in the cloud since it broke today after 16:00 and before that worked just fine.

UPDATE: I managed to fix this by updating the awscliv2 to the latest version, doing aws sso logout then aws sso login and then it worked. So might be something to do with token timings, and not CDK per se. Still, strange. AWS CLI commands worked just fine.

[megakoresh]

Updating: this seems to have something to do with timing. This happens if the session expires. Once it does, CDK will break and no amount of logging out and back in will help. Only waiting seems to do the trick. I have also tried clearing context, but still nothing. After waiting for hours, and then logging out and back in the problem goes away until the next day, but that is hardly a "solution". Is anyone looking into this?

[sturman]

I think the problem is in web_identity_token_file.
In my case with AWS_PROFILE=oidc environment variable set and the following content in ~/.aws/config file

[profile oidc]
role_arn=arn:aws:iam::012345678901:role/gitlab-deploy
web_identity_token_file=/tmp/web_identity_token

command aws sts get-caller-identity works as expected.
But npx cdk diff -vv fails with

...
[16:42:45] Determining if we're on an EC2 instance.
[16:42:45] Does not look like an EC2 instance.
[16:42:45] Toolkit stack: CDKToolkit
[16:42:45] Setting "CDK_DEFAULT_REGION" environment variable to eu-central-1
[16:42:45] Resolving default credentials
[16:42:45] Unable to determine the default AWS account (ProcessCredentialsProviderFailure): Profile oidc did not include credential process
[16:42:45] context: {
...

So I ended up assuming role and setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables with command

export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
$(aws sts assume-role-with-web-identity
--role-arn ${ROLE_ARN}
--role-session-name "GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
--web-identity-token ${GITLAB_OIDC_TOKEN}
--duration-seconds 3600
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
--output text))

[TheRealAmazonKendra]

I'm pretty sure the issue here is that our current credential chain does not support web_identity_token_file inside an ini file, so the credential chain here fails on profileCredentials because of the presence of web_identity_token_file and then again on SsoCredentials because none are setup, as expected. The last credential provider is ProcessCredentials, which will fail here, also as expected. Since it is the last in the chain, that's the error that actually gets surfaced, which is super misleading.

We are currently in the process of upgrading from sdkv2 to sdkv3, at which point web_identity_token_file will be supported and you should no longer experience this error. You can follow the progress in #31702, if you'd like.