(pipelines): limit assets concurrency

[rix0rrr]

Description

There are limits to how many CodeBuild projects can run in parallel (it's either 5 or 20), and asset publishing is liable to hit those.

Rather than queueing builds, any builds over the limit are just outright failed.

The CodePipeline generator should take this into account when generating asset publishing stages.

Use Case

Hitting CodeBuild limit.

Proposed Solution

Property to limit asset concurrency.

Other information

No response

Acknowledge

• I may be able to implement this feature request
• This feature might incur a breaking change

[gasser-mamdouh]

I'm really interested to work on this task, can I work on it?

if I can, for example: if a user has a max of 5 parallel runs, and has 8 assets, I want to make the assets stage have 2 steps the first step have from FileAsset1 to FileAsset5, and the second step have from FileAsset6 to FileAsset8, that's the expected behaviour right?

[kadrach]

Does the pipeline builder not already do this?

aws-cdk/packages/@aws-cdk/pipelines/lib/codepipeline/codepipeline.ts

Lines 410 to 413 in 588ddf1

	const chunks = chunkTranches(50, stageNode.sortedLeaves());
	const actionsOverflowStage = chunks.length > 1;
	for (const [i, tranches] of enumerate(chunks)) {
	const stageName = actionsOverflowStage ? `${stageNode.id}.${i + 1}` : stageNode.id;

The default quota for concurrent CodeBuild jobs is 60 per region, and the maximum number of parallel actions in a CodePipeline stage is 50.

[volkanunsal]

Is there a workaround? We have more than 20 lambdas in this service, and I cannot make any headway because of this issue. Is it possible to adjust this limit?

@rix0rrr

[ilgiznurgaliev]

any updates on this?

[bchex]

I am getting an error with our CDK CodePipeline that says Error calling startBuild: Cannot have more than 20 builds in queue for the account.

There is a service limit for "Concurrently running builds", but it's set to 60 by default. I've already had that limit increased by support twice, and still getting the errors. The AWS Docs are relatively clear, but my experience shows that CodePipeline isn't honoring the new limits. I'm guessing that the limits for ComputeType (we're using Large) are usurping the account limits?

I think this issue will solve our problem but it's not clear. Would really like to see some traction on this.

[NiallJoeMaher]

Is there any workaround for this? I've asked for a quota increase but would be nice to have a solution while waiting around if there is one.
Thanks! 🙏

[pfried]

I never actually had this issue despite having a rather large pipeline (large instance) with a lot of lambda assets (split across 2 stages). I came back from vacation and did a deploy and voila: "Error calling startBuild: Cannot have more than 10 builds in queue for the account".

All erroring with: Error calling startBuild: Cannot have more than 10 builds in queue for the account (Service: AWSCodeBuild; Status Code: 400; Error Code: AccountLimitExceededException; Request ID: c61981e6-8b35-471c-be6e-3e62599682c7; Proxy: null)

Maybe we never had that many changes (I doubt) or something changed recently

[nick-kang]

I'm hitting this error too. For new accounts, Codebuild concurrency is set to 1. That means you cannot even use CDK Pipeline if you have File Assets and Image Assets as it'll create 2 concurrent builds and throw an error.

I don't recall having this problem for new accounts in the past. Don't know if something changed in the way Codebuild enforces concurrency limits or if Codepipeline changed the way they queue builds. This is a major blocker and I'm not aware of any workarounds.

[DankersW]

Same error here.
Error calling startBuild: Cannot have more than 1 builds in queue for the account (Service: AWSCodeBuild; Status Code: 400; Error Code: AccountLimitExceededException; Request ID: 2cd62e77-6dc1-4c51-9480-9243fba52c91; Proxy: null)

Nothing changed on our configuration side. Pipelines that have many lambdas do not run any longer.

[vgjenks]

Having the same problem. Only allowing 1 build and forcing you to request a limit increase through AWS, is sort of ridiculous. Who can build even a small, mostly pointless pipeline with only 1 concurrent build!? At the very least, we should be able to configure this and accept the billing consequences. It renders CodePiplelines useless to me, at this point.

[mouhassan]

The CodePipeline construct has a property (publishAssetsInParallel) that should help avoiding this problem.

Setting it to false should result in a single CodeBuild project for publishing all assets. This avoids the concurrency problem if the speed of publishing assets is not a concern.

[nick-kang]

I believe that still creates two Codebuild Projects: one for Lamnda assets and another for container assets.

[byongwu]

Sorry about the inconvenience. I just checked that CodeBuild quota has not been increased.
Please ignore what I wrote below.

---

The CodePipeline construct has a property (publishAssetsInParallel) that should help avoiding this problem.

Setting it to false should result in a single CodeBuild project for publishing all assets. This avoids the concurrency problem if the speed of publishing assets is not a concern.

Maybe, it's only me but I am still having failure like this.

Error calling startBuild: Cannot have more than 1 builds in queue for the account (Service: AWSCodeBuild; Status Code: 400; Error Code: AccountLimitExceededException; Request ID: cfeca6db-082d-4b33-a4fb-938c72c3d3ff; Proxy: null)

Also, I applied the publishAssetsInParallel option like this.

    const pipeline = new pipelines.CodePipeline(this, "Pipeline", {
      pipelineName: props.codepipelineName,
      synth: steps,
      publishAssetsInParallel: false,
    });