(pipelines): could not assume role in another account

[myashchenko]

Describe the bug

I have a dedicated AWS account for running pipeline (let's call it account-1) and another AWS account where all resources being deployed by pipeline (account-2).

Everything worked just perfectly till today: I added a new stack to be deployed to account-2 and pushed changes to remote git repo.

Expected Behavior

A new stack is deployed successfully to account-2 via CodePipeline. No errors in pipeline shown.

Current Behavior

The Build step of CodePipeline started failing with the following error:

[Error at /PipelineStack/DevStage/NetworkStack] Could not assume role in target account using current credentials (which are for account account-1) User: arn:aws:sts::account-1:assumed-role/PipelineStack-AppNamePipelineBuil-XXXXXXX/AWSCodeBuild-uuid is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::account-2:role/cdk-xxxx-lookup-role-account-2-us-west-2 . Please make sure that this role exists in the account. If it doesn't exist, (re)-bootstrap the environment with the right '--trust', using the latest version of the CDK CLI.

Reproduction Steps

export class PipelineStack extends Stack {

    constructor(scope: Construct) {
        super(scope, 'PipelineStack', {
            env: CI_ENV
        });

        const synth = new SynthesizePipelineBuildStep(GITHUB_CONNECTION)
        const pipeline = new CodePipeline(this, "Pipeline", {
            pipelineName: "Pipeline",
            crossAccountKeys: true,
            synth: synth,
        });

        const devStage = new DevStage(this);
        pipeline.addStage(devStage)
    }
}

export class DevStage extends Stage {

    public readonly networkStack: NetworkStack
    public readonly lambdaFunctionStack: LambdaFunctionStack

    constructor(scope: Construct) {
        super(scope, 'DevStage', {
            env: DEV_ENV
        });

        this.networkStack = new NetworkStack(this)
        this.lambdaFunctionStack = new LambdaFunctionStack(this)
    }
}

// this is a new stack
export class NetworkStack extends Stack {

    public readonly vpc: IVpc

    constructor(scope: Construct, props?: StackProps) {
        super(scope, 'NetworkStack', props);

        this.vpc = Vpc.fromLookup(this, 'TestVPC', {
            vpcId: 'vpc-xxxxxxxx'
        })
    }
}

// this is an old stack which had been previously deployed successfully, before I added NetworkStack
export class LambdaFunctionStack extends Stack {

    constructor(scope: Construct, props?: StackProps) {
        super(scope, 'LambdaFunctionStack', props);

        const lambda = new AwesomeLambdaFunction(this)
    }
}

Possible Solution

I went to arn:aws:sts::account-1:assumed-role/PipelineStack-AppNamePipelineBuil-XXXXXXX role and edited it manually by adding AssumeRole on arn:aws:iam::account-2:role/cdk-xxxx-lookup-role-account-2-us-west-2. After that pipeline has completed successfully.

Although I don't feel like I should have done manual update of the role. All the necessary permissions should be granted automatically when running cdk bootstrap

Additional Information/Context

I tried re-running cdk bootstrap (although I did this already when initially deploying the pipeline):

cdk bootstrap --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://account-1/us-west-2

cdk bootstrap --trust account-1 --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://account-2/us-west-2

But it didn't help.

CDK CLI Version

2.18.0

Framework Version

No response

Node.js Version

17.7.1

OS

MacOS

Language

Typescript

Language Version

No response

Other information

No response

[rix0rrr]

You probably added code which requires a context lookup, without commiting the corresponding update to cdk.context.json.

CDK will then try and do the lookup at build time.

• You should never want the commit to be done at synth time. If you do want that, read this: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.pipelines-readme.html#context-lookups
• Preferred method: synth on your development machine and commit the changes to cdk.context.json.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[HeskethGD]

You probably added code which requires a context lookup, without commiting the corresponding update to cdk.context.json.

CDK will then try and do the lookup at build time.

• You should never want the commit to be done at synth time. If you do want that, read this: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.pipelines-readme.html#context-lookups
• Preferred method: synth on your development machine and commit the changes to cdk.context.json.

Would you be able to elaborate here a little as the docs seem a bit confusing to me. We have hit this issue when using cdk pipelines as we are trying to reference a target bucket in another account from the pipeline account to deploy a react app into it. I don't understand these sentences from that link: "This will make sure your synthesized infrastructure is consistent and repeatable. If you do not commit cdk.context.json, the results of the lookups may suddenly be different in unexpected ways, and even produce results that cannot be deployed or will cause data loss. " ... "If you want to use lookups directly from the pipeline, you either need to accept the risk of nondeterminism, or make sure you save and load the cdk.context.json file somewhere between synth runs. " What is nondeterminism and why would things be changing in unexpected ways if they were looked up at runtime in the deployment pipeline? If I added the permissions in the role as the op did so that it could look up the bucket, could you give me an example of how it might go wrong in unexpected ways?