(pipelines): pipeline with action to deploy stack with pipeline with cross region support generates invalid cross region stack

[rehos]

Describe the bug

We have a CDK application deploys a pipeline (call it A) into an account (000000000000) in eu-central-1 which in turn deploys another pipeline in a tenant account (111111111111) in the same region. The pipeline (call it T) in the tenant account has a cross region deployment and requires a cross region support stack. Because of that the application A should also deploy a cross region support stack to deploy the cross region support stack for the tenant account.

After running npx cdk synth you can see the problem. Open the file cdk.out/cross-region-stack-000000000000:us-east-1.template.json. The KMS Key policy and the bucket resource policy reference roles in eu-central-1 instead of us-east-1. This results in permissions error during deployment.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Generate the cross region support stack the references cdk roles corresponding to the region. The action to deploy the support stack should als reference the correct roles.

Current Behavior

It generates a cross region support stack that references the region of the pipeline itself. The action to deploy the support stack references the roles corresponding to the region of the pipeline itself..

Reproduction Steps

I have created a minimal CDK application to reproduce the error.

See https://github.com/rehos/aws-cdk-pipelines-cross-region-issue

Possible Solution

Cross region stacks should only reference CDK roles for the target region

Additional Information/Context

No response

CDK CLI Version

2.1000.2

Framework Version

No response

Node.js Version

22.14.0

OS

macOS 15.3.1

Language

TypeScript

Language Version

Typescript (5.6.3)

Other information

At the moment we use escape hatches to fix the roles referenced in the action in the pipeline and add the correct role to the key policy and bucket policy

[pahud]

Looking into it.

[pahud]

Let me clarify your use case using a mermaid as below. Is this correct?

graph TD
    A[Account 000000000000<br>eu-central-1] -->|Deploys| B[Pipeline A]
    B -->|Deploys| C[Account 111111111111<br>eu-central-1]
    C --> D[Pipeline T]
    D -->|Deploys| E[MyTenantApplicationEUStack<br>eu-central-1]
    D -->|Cross-region<br>Deploys| F[MyTenantApplicationUSStack<br>us-east-1]
    F -.-> G[Cross-region Support Stack<br>Incorrectly references eu-central-1 roles]
    
    %% Dark theme styles
    classDef default fill:#2d2d2d,stroke:#666,color:#fff
    classDef highlight fill:#44475a,stroke:#6272a4,color:#f8f8f2
    classDef error stroke:#ff5555,stroke-width:2px,stroke-dasharray: 5 5,color:#ff5555
    
    style A fill:#bd93f9,stroke:#6272a4
    style C fill:#8be9fd,stroke:#50fa7b
    style F fill:#ffb86c,stroke:#ff79c6
    style G error

Loading

Now, to help isolate this issue, could you please try manually deploying both stacks separately to see if the cross-region support stack references are correct?

For example

# Deploy EU stack
cdk deploy MyTenantApplicationEUStack --region eu-central-1

# Deploy US stack
cdk deploy MyTenantApplicationUSStack --region us-east-1

This test will help us determine if:

1. The issue is specific to pipeline deployment
2. The problem exists in general CDK cross-region support stack generation

After deployment, please:

1. Check the generated CloudFormation templates in cdk.out/

cat cdk.out/MyTenantApplicationUSStack.template.json | jq '.Resources'

2. Verify the IAM roles and policies in both regions
3. Compare the manual deployment templates with pipeline-generated ones

This information will help us better understand whether the issue is pipeline-specific or a broader CDK cross-region support stack generation problem.

Thank you for your help in investigating this issue!

[rehos]

@pahud the diagram is correct. I will provide you with the requested information.

[rehos]

@pahud I have run the deploy commands (using the correct stack ids) and the stacks are deployed without any issues. The cdk.out folder is exactly the same for both cdk deploy commands and cdk synth (I removed the cdk.out folder before running each command). That there is no difference is also what I expected. I have added the full cdk.out folders to my repository in the misc folder (I replaced account ids in the cdk.out content with dummy ids 000000000000 and 111111111111).

The folder misc/cdk.out-synth contains the cdk.out content of the command:

cdk synth

The folder misc/cdk.out-deploy-eustack contains the cdk.out content of the command:

cdk deploy MyApplicationStack/MyTenantStage/MyTenantPipeline/MyTenantApplicationStage/MyTenantApplicationEUStack --region eu-central-1

The folder misc/cdk.out-deploy-usstack contains the cdk.out content of the command:

cdk deploy MyApplicationStack/MyTenantStage/MyTenantPipeline/MyTenantApplicationStage/MyTenantApplicationUSStack --region us-east-1

[rehos]

With respect to the diagram I said it was correct, but it is not (i missed the note with respect the cross region stack. The cross region support stack required by pipeline T requires a cross region support stack for pipeline A. And that cross region support stack references the wrong regions