[cdk-pipelines] Pipeline template grows very fast reaching the 460,800 bytes limit from CloudFormation #9225

JFox · 2020-07-23T08:28:13Z

❓ General Issue

The Question

After adding around 14 stacks (25 counting nested stacks) to a single Pipeline Application (Stage) the pipeline references 27 assets and the template grows above CloudFormation's 460,800 bytes limit, blocking the pipeline and making it unusable.

Is there a way to avoid this by compressing the generated template or by splitting it somehow? I'm facing this on a staging stage and production is way bigger

My suspicion is that the assets make the template grow too fast as each of them generates a build action with the whole chain of roles, policies, etc.

Environment

CLI Version : 1.51.0 (build 8c2d53c)
Framework Version: 1.51.0 (build 8c2d53c)
Node.js Version: v12.18.0
OS : aws/codebuild/standard:4.0
Language (Version): TypeScript (3.9.5)

njlynch · 2020-07-23T11:11:01Z

This sounds related to #9066, but possibly broader in scope.

srethira · 2020-07-24T00:25:03Z

I ran into same exact problem. Here is the github issue I created - #9237. Is there any workaround for this problem, until further fix to it ?

This change combines the two roles (and policies) per pipeline asset into a single role + policy for all assets in a pipeline. In a small pipeline with only 3 assets, this reduced the overall template size by a third. With a pipeline stack with 30 assets, this change reduces the template size from 363111 bytes to 190223 bytes. Tested on a cross-account pipeline to verify permissions were retained. fixes #9066 mitigates #9225 mitigates #9237

…ies (#9243) This change combines the two roles (and policies) per pipeline asset into a single role + policy for all assets in a pipeline. In a small pipeline with only 3 assets, this reduced the overall template size by a third. With a pipeline stack with 30 assets, this change reduces the template size from 363111 bytes to 190223 bytes. Tested on a cross-account pipeline to verify permissions were retained. fixes #9066 mitigates #9225 mitigates #9237 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

njlynch · 2020-07-28T09:23:15Z

Hi @JFox (and @srethira) --

I just merged a change (#9243) that should help with this issue. Given your description (14 stacks, 25 including nested stacks, and 27 assets), I'm not 100% sure that it will be enough, but it should significantly help. (With even just 10 assets, this change cut the template size in half). The fix will go out in the next release; once it does, I would appreciate any feedback on if it was sufficient to solve the problem for you, or if more work is necessary.

Thanks!

JFox · 2020-07-28T09:36:19Z

Thank you so much for taking care of this and for doing it so quickly.

I would keep a very close eye and give it a try as soon as it is released and get back to you with specific numbers.

srethira · 2020-07-28T12:54:53Z

@njlynch Thanks for providing the fix in a short period of time. Do you know when this will be released?

njlynch · 2020-07-28T13:31:15Z

@njlynch Thanks for providing the fix in a short period of time. Do you know when this will be released?

Soon! We typically release once a week.

seawatts · 2020-07-28T20:09:55Z

Hi @JFox (and @srethira) --

I just merged a change (#9243) that should help with this issue. Given your description (14 stacks, 25 including nested stacks, and 27 assets), I'm not 100% sure that it will be enough, but it should significantly help. (With even just 10 assets, this change cut the template size in half). The fix will go out in the next release; once it does, I would appreciate any feedback on if it was sufficient to solve the problem for you, or if more work is necessary.

Thanks!

I've got about 10 stacks with around 30 - 40 assets, with more to come. So we'll see. 🤞 I'll keep you updated.
I was thinking I could just create multiple pipelines / apps, however, because I have a shared stack that all the other stacks rely on, that solution doesn't work.

seawatts · 2020-07-29T01:10:27Z

Just tried out the latest fix (#9243) with v1.55.0 But now I am seeing a new error 😦

Maximum policy size of 10240 bytes exceeded for role PipelineStack-PipelineAssetsRole9B011B83-1UOLXJQTUXXXX

cc: @njlynch

njlynch · 2020-07-29T10:45:08Z

@seawatts - Thanks for the quick turn-around and feedback. We'll investigate the new issue with the policy size.

JFox · 2020-07-29T12:12:35Z

As a first test, we went by deploying the new version it to a fresh new account and it seems to work now, we are not getting the policy limit mentioned by @seawatts. However I won't totally discard it. I will post back once we run it in the pipeline that is nearly below the 460,800 bytes limit

JFox · 2020-07-29T12:32:23Z

I correct myself, we also hit the Maximum policy size of 10240 bytes exceeded

srethira · 2020-07-29T18:38:55Z

Hi @JFox (and @srethira) --
I just merged a change (#9243) that should help with this issue. Given your description (14 stacks, 25 including nested stacks, and 27 assets), I'm not 100% sure that it will be enough, but it should significantly help. (With even just 10 assets, this change cut the template size in half). The fix will go out in the next release; once it does, I would appreciate any feedback on if it was sufficient to solve the problem for you, or if more work is necessary.
Thanks!

I've got about 10 stacks with around 30 - 40 assets, with more to come. So we'll see. 🤞 I'll keep you updated.
I was thinking I could just create multiple pipelines / apps, however, because I have a shared stack that all the other stacks rely on, that solution doesn't work.

That's my thought process to overcome to issue. Unfortunately, I'm in the same boat as you're on with stack dependency.

njlynch · 2020-07-30T12:42:39Z

#9333 has been merged and will go out in the next release. I can create a pipeline with 50 assets successfully with this new approach. After that, we run into issues with limits on the number of actions in a single Pipeline stage. #9353 has been filed to track support for > 50 assets.

JFox · 2020-07-30T12:48:55Z

#9333 has been merged and will go out in the next release. I can create a pipeline with 50 assets successfully with this new approach. After that, we run into issues with limits on the number of actions in a single Pipeline stage. #9353 has been filed to track support for > 50 assets.

Great news! Thanks again for getting it done so fast

nonken · 2020-07-30T13:00:47Z

Super awesome!!! 🙏🏻

njlynch · 2020-08-03T10:10:05Z

CDK 1.56.0 was released on Friday; please do let us know if this solved the issue for you.

srethira · 2020-08-03T16:44:46Z

@njlynch I'm possibly running into #9353

71/101 | 4:35:41 PM | UPDATE_FAILED | AWS::CodePipeline::Pipeline | Pipeline/Pipeline (Pipeline9850B417) Pipeline stage 'Assets' has too many actions. There can only be up to 50 actions in a pipeline stage (Service: AWSCodePipeline; Status Code: 400; Error Code: InvalidStageDeclarationException; Request ID: 9304128e-3fa6-4793-8079-2525422cd668)

JFox · 2020-08-10T11:39:26Z

We deployed version 1.56.0 successfully with 31 assets being generated in the Assets stage. I guess this will buy us enough time for the root cause to be addressed. Thanks @njlynch

njlynch · 2020-08-10T11:40:45Z

Thanks for confirming, @JFox. Closing this out and tracking the remaining scaling issue (50 assets/actions) in #9353.

…ies (aws#9243) This change combines the two roles (and policies) per pipeline asset into a single role + policy for all assets in a pipeline. In a small pipeline with only 3 assets, this reduced the overall template size by a third. With a pipeline stack with 30 assets, this change reduces the template size from 363111 bytes to 190223 bytes. Tested on a cross-account pipeline to verify permissions were retained. fixes aws#9066 mitigates aws#9225 mitigates aws#9237 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

JFox added guidance Question that needs advice or information. needs-triage This issue or PR still needs to be triaged. labels Jul 23, 2020

github-actions bot assigned ericzbeard Jul 23, 2020

github-actions bot added the @aws-cdk/pipelines CDK Pipelines library label Jul 23, 2020

ericzbeard assigned rix0rrr and njlynch Jul 23, 2020

ericzbeard removed the needs-triage This issue or PR still needs to be triaged. label Jul 23, 2020

njlynch mentioned this issue Jul 24, 2020

fix(pipelines): Reduce template size by combining IAM roles and policies #9243

Merged

njlynch mentioned this issue Jul 28, 2020

[cdk pipelines] CDK - Issue with multiple Stacks within a Stage #9237

Closed

seawatts mentioned this issue Jul 29, 2020

[@aws-cdk/pipelines] Maximum policy size of 10240 bytes exceeded for role #9316

Closed

ericzbeard removed their assignment Aug 3, 2020

rix0rrr added bug This issue is a bug. effort/medium Medium work item – several days of effort p1 guidance Question that needs advice or information. and removed guidance Question that needs advice or information. labels Aug 4, 2020

njlynch closed this as completed Aug 10, 2020

NGL321 mentioned this issue Mar 8, 2022

Tracking: Policy-generation creates oversized templates #19276

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cdk-pipelines] Pipeline template grows very fast reaching the 460,800 bytes limit from CloudFormation #9225

[cdk-pipelines] Pipeline template grows very fast reaching the 460,800 bytes limit from CloudFormation #9225

JFox commented Jul 23, 2020 •

edited

Loading

njlynch commented Jul 23, 2020

srethira commented Jul 24, 2020

njlynch commented Jul 28, 2020

JFox commented Jul 28, 2020 •

edited

Loading

srethira commented Jul 28, 2020

njlynch commented Jul 28, 2020

seawatts commented Jul 28, 2020

seawatts commented Jul 29, 2020 •

edited

Loading

njlynch commented Jul 29, 2020

JFox commented Jul 29, 2020

JFox commented Jul 29, 2020

srethira commented Jul 29, 2020

njlynch commented Jul 30, 2020

JFox commented Jul 30, 2020

nonken commented Jul 30, 2020

njlynch commented Aug 3, 2020

srethira commented Aug 3, 2020

JFox commented Aug 10, 2020

njlynch commented Aug 10, 2020

[cdk-pipelines] Pipeline template grows very fast reaching the 460,800 bytes limit from CloudFormation #9225

[cdk-pipelines] Pipeline template grows very fast reaching the 460,800 bytes limit from CloudFormation #9225

Comments

JFox commented Jul 23, 2020 • edited Loading

❓ General Issue

The Question

Environment

njlynch commented Jul 23, 2020

srethira commented Jul 24, 2020

njlynch commented Jul 28, 2020

JFox commented Jul 28, 2020 • edited Loading

srethira commented Jul 28, 2020

njlynch commented Jul 28, 2020

seawatts commented Jul 28, 2020

seawatts commented Jul 29, 2020 • edited Loading

njlynch commented Jul 29, 2020

JFox commented Jul 29, 2020

JFox commented Jul 29, 2020

srethira commented Jul 29, 2020

njlynch commented Jul 30, 2020

JFox commented Jul 30, 2020

nonken commented Jul 30, 2020

njlynch commented Aug 3, 2020

srethira commented Aug 3, 2020

JFox commented Aug 10, 2020

njlynch commented Aug 10, 2020

JFox commented Jul 23, 2020 •

edited

Loading

JFox commented Jul 28, 2020 •

edited

Loading

seawatts commented Jul 29, 2020 •

edited

Loading