Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

codepipeline: x-account AND x-region deployments are missing stack dependencies #24050

Open
rix0rrr opened this issue Feb 7, 2023 · 0 comments
Labels
@aws-cdk/aws-codepipeline Related to AWS CodePipeline bug This issue is a bug. p1

Comments

@rix0rrr
Copy link
Contributor

rix0rrr commented Feb 7, 2023

Describe the bug

A cross-region AND cross-account deployment where CodePipeline generates the support stacks for replication buckets and the support stacks for cross-account roles (which is the ideal situation), fails to deploy out of the box.

The cross-account and cross-region resources are created in separate stacks that don't have a dependency between them, but the resources on those stacks have bidirectional dependencies on each other:

  • The Bucket and Key policies reference the Action Role
  • The Role policy references the Bucket and Key (and deployment resources like CodeDeploy Deployment Groups)

Normally this wouldn't deploy, but we generate hard-coded resource names for the resources involved so that we can formulate policies anyway without having to have bidirectional cross-stack references.


The only order in which this deployment works is if we deploy the account stack (with the Role) before the replication stack (with the Bucket and Key), but there is no dependency between these stacks, so a naive cdk deploy may pick the wrong order and fail to deploy properly.


This is a tricky area to work in, as people may have built all kinds of elaborate constructions of stacks and resources and referenced resources, and anything that adds more dependencies is at risk of producing a cyclic dependency.

We are piloting a patchy fix right now for ECS CodeDeploy, which should be copied to all deployment actions if successful.

Expected Behavior

A cdk deploy should deploy in the right order.

Current Behavior

The replication bucket stack deploys before the role stack, causing a deployment failure.

Related issues

@rix0rrr rix0rrr added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 7, 2023
@github-actions github-actions bot added the @aws-cdk/aws-codepipeline Related to AWS CodePipeline label Feb 7, 2023
@rix0rrr rix0rrr added the p1 label Feb 7, 2023
rix0rrr added a commit that referenced this issue Feb 7, 2023
…k dependency

In the case of a cross-account AND cross-region ECS deployment, a
dependency between the support stacks that is necessary is missing.

This cannot be fixed globally. Because the resources reference each
other bidirectionally (user -> bucket, bucket -> user), the only way to
fix this is to do it locally: in the CodePipeline module, where we can
have the knowledge that we use generated names and that everything will
work out if we deploy the role before the bucket.

All CodePipeline Actions must have this fix eventually, but since
people may have crazy stack setups in which addition of this dependency
may introduce a cyclic dependency (breaking the synth), we're rolling
this fix out with limited blast radius.

Follow-up in #24050, and suggest a good clean-up in #24051.
@peterwoodworth peterwoodworth removed the needs-triage This issue or PR still needs to be triaged. label Feb 7, 2023
mergify bot pushed a commit that referenced this issue Feb 10, 2023
…ncy (#24053)

In the case of a cross-account AND cross-region ECS deployment, a dependency between the support stacks that is necessary is missing.

This cannot be fixed globally. Because the resources reference each other bidirectionally (user -> bucket, bucket -> user), the only way to fix this is to do it locally: in the CodePipeline module, where we can have the knowledge that we use generated names and that everything will work out if we deploy the role before the bucket.

All CodePipeline Actions must have this fix eventually, but since people may have crazy stack setups in which addition of this dependency may introduce a cyclic dependency (breaking the synth), we're rolling this fix out with limited blast radius.

Follow-up in #24050, and suggest a good clean-up in #24051.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-codepipeline Related to AWS CodePipeline bug This issue is a bug. p1
Projects
None yet
Development

No branches or pull requests

2 participants