(pipelines): Allow re-generating CloudFormation changeset after failure

[blimmer]

As of today, I couldn't find a way to re-run an application stage deployment if application of the CloudFormation changeset failed. We can kick off the pipeline again by committing a dummy/no-op commit to the repository, but this creates unnecessary cruft in the SCM history.

Use Case

Consider a CloudFormation template that could not apply for some reason. In my case, it was because of a naming conflict with another CloudFormation stack that needed to be destroyed.

The CodePipeline user interface looks like this after the failure:

So, I took a look at the failure and noticed the naming conflict. I went and destroyed the other CDK stack and wanted to re-run the deployment. However, when I clicked the "Retry" button in the UI, the "Deploy" stage failed with this error message:

ChangeSet [PipelineChange] does not exist (Service: AmazonCloudFormation; Status Code: 404; Error Code: ChangeSetNotFound; Request ID: <redacted>)

Indeed, in the CloudFormation user interface, the changeset was not present:

The problem appears to be because there's no way to retry an entire stage in CodePipeline, including steps that previously succeeded. According to the aws cli docs (aws codepipeline retry-stage-execution help):

--retry-mode (string)
  The scope of the retry attempt. Currently, the only supported  value
  is FAILED_ACTIONS.

  Possible values:

  o FAILED_ACTIONS

Because I didn't have a way to rerun the .Prepare step (which produces the changeset), I was kind of stuck and had to revert to pushing a no-op commit to force the step to re-run.

Proposed Solution

Since this appears to be a CodePipeline limitation, maybe a custom "step" is needed? Like maybe an SNS topic or something to kick off the deployment again? Or maybe a better solution would be to +1 feature requests on the CodePipeline side to allow "Retry (including succeeded)" behavior?

Other

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[blimmer]

Is it possible that this is only an issue with the initial deploy of the stack? Potentially similar to this: #901?

[blimmer]

I also just found this command to re-trigger a pipeline from the CLI - maybe it's just not exposed via the UI?

aws codepipeline start-pipeline-execution

Perhaps this is a good workaround.

[rix0rrr]

Or maybe a better solution would be to +1 feature requests on the CodePipeline side to allow "Retry (including succeeded)" behavior?

I would agree that a better solution would be to petition to fix the root cause, rather than try to work around it.

There should be a button at the top-right of your pipeline though that says "Release Change", which does the same as starting the execution using the CLI.

Not actionable for CDK. Closing.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[openwebsolns]

I also just found this command to re-trigger a pipeline from the CLI - maybe it's just not exposed via the UI?

aws codepipeline start-pipeline-execution

Perhaps this is a good workaround.

This is available through the UI by clicking the "Release change" button.

[openwebsolns]

This is still a problem. If you have a large pipeline, "releasing a change" would require deploying through all the previous stages before you can retry the failed stage. Is there a creative workaround by which we can "replay" a changeset with the same name (PipelineChange) so that the CodePipeline retry works? Does CDK provide the ability to just create the changeset?

[fab-mindflow]

Indeed, we also have a large CDK deployment and this forces us to build everything again and again. Any workaround?

[XTEKiyZRLr8UsiQN]

Indeed, we also have a large CDK deployment and this forces us to build everything again and again. Any workaround?

Hi, a workaround for this is to actually disable change sets. They are auto enabled but based on https://github.com/aws/aws-cdk/pull/21619/files you should be able to remove them. I have had success with this and can now retry my stages. Hope this helps.

https://github.com/aws/aws-cdk/pull/21619/files