Skip to content

fix(cicd): upload pipeline template to S3 bucket to prevent hitting size limit#2200

Merged
mergify[bot] merged 10 commits intoaws:mainlinefrom
huanjani:template-upload
Apr 23, 2021
Merged

fix(cicd): upload pipeline template to S3 bucket to prevent hitting size limit#2200
mergify[bot] merged 10 commits intoaws:mainlinefrom
huanjani:template-upload

Conversation

@huanjani
Copy link
Copy Markdown
Contributor

@huanjani huanjani commented Apr 21, 2021

The maximum size of a Cloudformation template body is 51,200 bytes. (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html)

When users have multiple services deployed to multiple environments and they want to build a pipeline, they risk hitting this limit.

By uploading the template to an S3 bucket, then passing the template URL to the ChangeSet, we take advantage of S3's multipart upload functionality (https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) and circumvent the size limit.

The size limit for a template passed in via URL is 460,800 bytes. (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloudformation-stackset.html)

Fixes #2068.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@huanjani huanjani requested a review from a team as a code owner April 21, 2021 21:46
@huanjani huanjani requested a review from bvtujo April 21, 2021 21:46
}

// Execute creates a new pipeline or updates the current pipeline if it already exists.
func (o *updatePipelineOpts) Execute() error {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: there are no changes to Execute()-- in the first commit I just moved this method higher up in the file.

Copy link
Copy Markdown
Contributor

@SoManyHs SoManyHs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fabulous! ✨ had a few code style questions, but nothing blocking :shipit:

Comment thread internal/pkg/aws/cloudformation/changeset.go Outdated
Comment thread internal/pkg/deploy/cloudformation/pipeline.go
Comment thread internal/pkg/deploy/cloudformation/pipeline_test.go
@efekarakus efekarakus added the do-not-merge Pull requests that mergify shouldn't merge until the requester allows it. label Apr 22, 2021
Copy link
Copy Markdown
Contributor

@efekarakus efekarakus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Just a tiny request to remove the conditional from the the cloudformation pkg

Comment thread internal/pkg/aws/cloudformation/changeset.go Outdated
Comment thread internal/pkg/aws/cloudformation/changeset.go Outdated
Comment thread internal/pkg/aws/s3/s3.go Outdated
Comment thread internal/pkg/deploy/cloudformation/pipeline.go Outdated
@efekarakus efekarakus removed the do-not-merge Pull requests that mergify shouldn't merge until the requester allows it. label Apr 23, 2021
@mergify mergify Bot merged commit a8f4b1b into aws:mainline Apr 23, 2021
efekarakus added a commit to efekarakus/copilot-cli that referenced this pull request Apr 28, 2021
mergify Bot pushed a commit that referenced this pull request Apr 28, 2021
…hitting size limit (#2200)" (#2228)

This reverts commit a8f4b1b.

Unfortunately, we are not guaranteed to have an S3 bucket in the application's region unless the user ran `env init` for the same region as the application. Our integration tests caught this bug 🌈 ! We'll patch the earlier fix in a follow-up. 

_By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice._
huanjani added a commit that referenced this pull request Apr 30, 2021
…prevent hitting size limit (#2200)" (#2228)"

This reverts commit 0a3d7c3.
@huanjani huanjani deleted the template-upload branch May 4, 2021 21:05
thrau pushed a commit to localstack/copilot-cli-local that referenced this pull request Dec 9, 2022
…ize limit (aws#2200)

The maximum size of a Cloudformation template body is 51,200 bytes. (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html)

When users have multiple services deployed to multiple environments and they want to build a pipeline, they risk hitting this limit. 

By uploading the template to an S3 bucket, then passing the template URL to the ChangeSet, we take advantage of S3's multipart upload functionality (https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) and circumvent the size limit.

The size limit for a template passed in via URL is 460,800 bytes. (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloudformation-stackset.html)

Fixes aws#2068.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
thrau pushed a commit to localstack/copilot-cli-local that referenced this pull request Dec 9, 2022
…hitting size limit (aws#2200)" (aws#2228)

This reverts commit a8f4b1b.

Unfortunately, we are not guaranteed to have an S3 bucket in the application's region unless the user ran `env init` for the same region as the application. Our integration tests caught this bug 🌈 ! We'll patch the earlier fix in a follow-up. 

_By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice._
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

upload pipeline template to S3 bucket to handle larger capacities

4 participants