CloudFormation with ChangeSet - Task fails if there are no changes #97

JeffRausch · 2018-08-22T15:06:45Z

@steveataws
I'm not sure if the changes done for #91 broke the changes done for issue #28 but now if I deploy a stack that uses a change set and there are no changes I get an error and the release fails. I didn't notice any options that I could change other than the "Log warning during stack update..." that was added for issue #91 . There is a "Continue on error" option under Control Options but we don't want to do that in case there are other issues with the CF deployment.
I get the same error whether the "Log warning during stack update..." option is checked or not.

The deployment without a change set works as expected.

2018-08-21T12:36:43.7456643Z Waiting for change set Release-20180821-2 to be validated for stack CF-XXXXXXXXXX

2018-08-21T12:37:14.0340548Z Change set creation request failed with error: 'Stack CF-XXXXXXXXXX failed to create successfully from the change set Release-20180821-2. Error: 'Resource is not in the state changeSetCreateComplete'' Error: Stack CF-XXXXXXXXXX failed to create successfully from the change set Release-20180821-2. Error: 'Resource is not in the state changeSetCreateComplete'
2018-08-21T12:37:14.0361661Z at . (D:\VSTS_Worker01_work_tasks\CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99\1.1.3\CloudFormationCreateOrUpdateStack.js:9:23778)

2018-08-21T12:37:14.0415842Z ##[error]Error: Stack CF-XXXXXXXXXX failed to create successfully from the change set Release-20180821-2. Error: 'Resource is not in the state changeSetCreateComplete'
2018-08-21T12:37:14.0434437Z at next (native)
2018-08-21T12:37:14.0436376Z at o (D:\VSTS_Worker01_work_tasks\CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99\1.1.3\CloudFormationCreateOrUpdateStack.js:9:12374)
2018-08-21T12:37:14.0465109Z at process._tickDomainCallback (internal/process/next_tick.js:135:7)

stevejroberts · 2018-08-22T15:53:26Z

The changes were just to suppress the warning message, not skip it and throw, so it shouldn't have caused a regression but...famous last words. Taking a look now.

stevejroberts · 2018-08-22T18:03:44Z

@JeffRausch , I must be doing something different to your build as I can't get a build fail when no changes are present. I set up a pipeline that deployed one of the CloudFormation sample templates, then redeployed it, and I'm not seeing an issue.

The output below is from the default task settings with the 'log no-change warning' option toggled oin:

Checking existence for stack cfn-createStack-fixedCreds
Test for existence of stack cfn-createStack-fixedCreds returned error: 'ValidationError: Stack with id cfn-createStack-fixedCreds does not exist'.
Stack does not exist, switching to create stack mode
Testing to see if change set cfn-createStack-fixedCredsCS associated with stack cfn-createStack-fixedCreds exists
Test for existence of change set cfn-createStack-fixedCredsCS returned error: 'Stack [cfn-createStack-fixedCreds] does not exist'.
Uploading template file C:\TfsAgents\1\_work\37\s\cfntemplates\WordPress_Single_Instance.template to object WordPress_Single_Instance.template in bucket vsts-testing
Generated url to template: undefined
Loading template parameters from task definition
Successfully loaded template parameters
Setting capability CAPABILITY_IAM for stack
Setting capability CAPABILITY_NAMED_IAM for stack
Creating CREATE type change set cfn-createStack-fixedCredsCS
Waiting for change set cfn-createStack-fixedCredsCS to be validated for stack cfn-createStack-fixedCreds
Change set successfully validated
Executing change set cfn-createStack-fixedCredsCS cfn-createStack-fixedCreds
Waiting for stack cfn-createStack-fixedCreds to reach create complete status
Stack cfn-createStack-fixedCreds transitioned to create complete status
Stack cfn-createStack-fixedCreds processed successfully, stack ID arn:aws:cloudformation:us-west-2:...:stack/cfn-createStack-fixedCreds/9680f710-a632-11e8-b84d-06f917171b52
******************************************************************************
Finishing: Create stack
******************************************************************************
******************************************************************************
Starting: Redeploy with no changes
******************************************************************************
...
Checking existence for stack cfn-createStack-fixedCreds
Stack exists, switching to update stack mode
Testing to see if change set cfn-createStack-fixedCredsCS associated with stack cfn-createStack-fixedCreds exists
Test for existence of change set cfn-createStack-fixedCredsCS returned error: 'ChangeSet [cfn-createStack-fixedCredsCS] does not exist'.
Uploading template file C:\TfsAgents\1\_work\37\s\cfntemplates\WordPress_Single_Instance.template to object WordPress_Single_Instance.template in bucket vsts-testing
Generated url to template: undefined
Loading template parameters from task definition
Successfully loaded template parameters
Setting capability CAPABILITY_IAM for stack
Setting capability CAPABILITY_NAMED_IAM for stack
Creating UPDATE type change set cfn-createStack-fixedCredsCS
Waiting for change set cfn-createStack-fixedCredsCS to be validated for stack cfn-createStack-fixedCreds
WARNING: no changes were detected for the template or change set. The stack was not updated.
Stack cfn-createStack-fixedCreds processed successfully, stack ID arn:aws:cloudformation:us-west-2:...:stack/cfn-createStack-fixedCreds/9680f710-a632-11e8-b84d-06f917171b52
******************************************************************************
Finishing: Redeploy with no changes
******************************************************************************

The second output below is with the option toggled off - the warning disappears but the task completes successfully -

Stack does not exist, switching to create stack mode
Testing to see if change set cfn-createStack-fixedCredsCS associated with stack cfn-createStack-fixedCreds exists
Test for existence of change set cfn-createStack-fixedCredsCS returned error: 'Stack [cfn-createStack-fixedCreds] does not exist'.
Uploading template file C:\TfsAgents\1\_work\37\s\cfntemplates\WordPress_Single_Instance.template to object WordPress_Single_Instance.template in bucket vsts-testing
Generated url to template: undefined
Loading template parameters from task definition
Successfully loaded template parameters
Setting capability CAPABILITY_IAM for stack
Setting capability CAPABILITY_NAMED_IAM for stack
Creating CREATE type change set cfn-createStack-fixedCredsCS
Waiting for change set cfn-createStack-fixedCredsCS to be validated for stack cfn-createStack-fixedCreds
Change set successfully validated
Executing change set cfn-createStack-fixedCredsCS cfn-createStack-fixedCreds
Waiting for stack cfn-createStack-fixedCreds to reach create complete status
Stack cfn-createStack-fixedCreds transitioned to create complete status
Stack cfn-createStack-fixedCreds processed successfully, stack ID arn:aws:cloudformation:us-west-2:...:stack/cfn-createStack-fixedCreds/b33be530-a633-11e8-8820-0688a7bf983a
******************************************************************************
Finishing: Create stack
******************************************************************************
******************************************************************************
Starting: Redeploy with no changes
******************************************************************************
...
Checking existence for stack cfn-createStack-fixedCreds
Stack exists, switching to update stack mode
Testing to see if change set cfn-createStack-fixedCredsCS associated with stack cfn-createStack-fixedCreds exists
Test for existence of change set cfn-createStack-fixedCredsCS returned error: 'ChangeSet [cfn-createStack-fixedCredsCS] does not exist'.
Uploading template file C:\TfsAgents\1\_work\37\s\cfntemplates\WordPress_Single_Instance.template to object WordPress_Single_Instance.template in bucket vsts-testing
Generated url to template: undefined
Loading template parameters from task definition
Successfully loaded template parameters
Setting capability CAPABILITY_IAM for stack
Setting capability CAPABILITY_NAMED_IAM for stack
Creating UPDATE type change set cfn-createStack-fixedCredsCS
Waiting for change set cfn-createStack-fixedCredsCS to be validated for stack cfn-createStack-fixedCreds
Stack cfn-createStack-fixedCreds processed successfully, stack ID arn:aws:cloudformation:us-west-2:563688378412:stack/cfn-createStack-fixedCreds/b33be530-a633-11e8-8820-0688a7bf983a
******************************************************************************
Finishing: Redeploy with no changes
******************************************************************************

Can you share the settings you have on your Create/Update Stack task?

JeffRausch · 2018-08-22T18:19:30Z

Below is the JSON exported from that task. The only thing I see that may be different is I'm trying to update an already updated stack. So my stack status is "UPDATE_COMPLETE" instead of "CREATE_COMPLETE" as seen with a freshly created stack.
I also attached the debug log from the task.

{
	"environment": {},
	"taskId": "7ef7cdfa-aa45-42c5-93c8-{REDACTED}",
	"version": "1.*",
	"name": "Create/Update Stack: CF-$(DeploymentEnvironment)-CFModifyS3Event",
	"refName": "",
	"enabled": true,
	"alwaysRun": false,
	"continueOnError": false,
	"timeoutInMinutes": 0,
	"definitionType": "task",
	"overrideInputs": {},
	"condition": "succeeded()",
	"inputs": {
		"awsCredentials": "3ba942d5-{REDACTED}",
		"regionName": "us-east-1",
		"stackName": "CF-$(DeploymentEnvironment)-CFModifyS3Event",
		"templateSource": "file",
		"templateFile": "$(System.DefaultWorkingDirectory)/CloudFormation/CloudFormationLambdaTemplates/CloudFormation.ModifyS3Event.CF-Lambda-Template.json",
		"s3BucketName": "",
		"s3ObjectKey": "",
		"templateUrl": "",
		"templateParametersSource": "inline",
		"templateParametersFile": "",
		"templateParameters": "[{\"ParameterKey\":\"DeploymentEnvironment\", \"ParameterValue\":\"$(DeploymentEnvironment)\"}]",
		"useChangeSet": "true",
		"changeSetName": "Release-$(release.releasename)",
		"description": "Change set for current release",
		"autoExecuteChangeSet": "true",
		"capabilityIAM": "false",
		"capabilityNamedIAM": "true",
		"roleARN": "",
		"resourceTypes": "",
		"notificationARNs": "",
		"tags": "",
		"monitorRollbackTriggers": "false",
		"monitoringTimeInMinutes": "0",
		"rollbackTriggerARNs": "",
		"onFailure": "ROLLBACK",
		"disableRollback": "false",
		"warnWhenNoWorkNeeded": "true",
		"outputVariable": "",
		"timeoutInMins": "60",
		"logRequest": "true",
		"logResponse": "true"
	}
}

JeffRausch_CFN_StackUpdateFailure.log

stevejroberts · 2018-08-22T21:45:27Z

Hmmm, updating my pipeline to do a create, update then another update didn't throw up. Something odd going on :-(

JeffRausch · 2018-08-23T12:49:38Z

Just to confirm, you Created the stack, then did a stack update with changes, then tried to do a stack update without changes. Other than that I can't think of anything else that may be different.

When I try to deploy it creates the change set. I see the change set in the AWS CloudFormation console and it finishes with a status of FAILED. If I go into that failed changeset I see the status as "FAILED - No updates are to be performed".

If I get time today I'll try to put together a simple test to reproduce it on my end. Maybe there's something odd with the deployment that I missed or possibly it depends on what I'm trying to deploy since it contains a lambda function or something.

stevejroberts · 2018-08-23T15:01:42Z

My sequence was create, update-no-changes, update-no-changes. I'll revise that second step to make a change and see what occurs.

stevejroberts · 2018-08-24T18:06:31Z

A sequence of create, update-with-change and update-with-no-change is also proving elusive to repro. Did you manage to get a simplified repro I could try?

JeffRausch · 2018-08-24T19:27:40Z

Sorry, I didn't have time yesterday. I just put a small sample together and was able to reproduce it. It was reproduced by creating the stack and then trying to update the stack using the same template with no changes - I did not need to update the stack with changes. Potentially the key is having a lambda function in the stack...?

"index.zip" contains the CloudFormation template and a small js file to test the lambda function create. I loaded the zip to a S3 location and then updated that location within the template.
"CFN Test Pipeline.zip" contains the full Release pipeline that I exported. I just changed names, replaced the company name with "{REDACTED}" and replaced most of the AWS Credential guid with "{REDACTED}".

index.zip
CFN Test Pipeline.zip

stevejroberts · 2018-08-27T17:42:07Z

Thanks, I'll give those a try.

stevejroberts · 2018-08-27T22:08:22Z

Ok, I think I see what's happening. If I fire off a pipeline with a create and then a no-op update, the build fails at the VSTS side with this error -

Creating UPDATE type change set test1cs2
Waiting for change set test1cs2 to be validated for stack test1
Change set creation request failed with error: 'Stack test1 failed to create successfully from the change set test1cs2. Error: 'Resource is not in the state changeSetCreateComplete'' Error: Stack test1 failed to create successfully from the change set test1cs2. Error: 'Resource is not in the state changeSetCreateComplete'
    at h.<anonymous> (C:\TfsAgents\1\_work\_tasks\CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99\1.1.4\CloudFormationCreateOrUpdateStack.js:10:25509)
    at next (native)
    at a (C:\TfsAgents\1\_work\_tasks\CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99\1.1.4\CloudFormationCreateOrUpdateStack.js:10:13806)
    at process._tickDomainCallback (internal/process/next_tick.js:135:7)
Error: Stack test1 failed to create successfully from the change set test1cs2. Error: 'Resource is not in the state changeSetCreateComplete'

However if I check in the console, the change set has the following message against it:

FAILED - No updates are to be performed.

I can trigger this even if I delete the change set (in the console) and then in my VSTS build just run an update step with no prior create.

I don't know yet if this is a change of behavior at the service side or a change of message - the current code is looking for a state of FAILED and a message 'The submitted information didn't contain changes' (this was the only way to detect no-op sadly). The text 'No updates are to be performed' was originally returned only if updating a stack without using a change set - so it's possible the service team have consolidated to one message.

I'm going to run some tests to check on either message for a change set update see if that works, then we can try and unblock you.

…, caused by change of service behavior

Fix for issue #97, no longer detecting 'no work done' on stack update

stevejroberts · 2018-08-29T19:55:06Z

@JeffRausch , I just pushed v1.1.5 to the marketplace that hopefully fixes the issue for you - I'll leave the issue open pending confirmation, feel free to close if there are no other related problems.

JeffRausch · 2018-08-30T15:16:35Z

It's working as expected now. Thanks!

stevejroberts added a commit that referenced this issue Aug 29, 2018

Change how we detect 'no work done' for a stack update to fix issue #97…

e90cd71

…, caused by change of service behavior

stevejroberts added a commit that referenced this issue Aug 29, 2018

Merge tag 'v1.1.5' into develop

39559e0

Fix for issue #97, no longer detecting 'no work done' on stack update

stevejroberts added the pending-confirmation label Aug 30, 2018

JeffRausch closed this as completed Aug 30, 2018

stevejroberts removed the pending-confirmation label Aug 31, 2018

RLThomaz mentioned this issue Nov 1, 2018

Inconsistent createChangeSet status for change sets without any changes aws/aws-sdk-js#2331

Closed

Janpot mentioned this issue May 27, 2020

no-fail-on-empty-changeset doesn't seem to work aws-actions/aws-cloudformation-github-deploy#14

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CloudFormation with ChangeSet - Task fails if there are no changes #97

CloudFormation with ChangeSet - Task fails if there are no changes #97

JeffRausch commented Aug 22, 2018

stevejroberts commented Aug 22, 2018 •

edited

stevejroberts commented Aug 22, 2018

JeffRausch commented Aug 22, 2018

stevejroberts commented Aug 22, 2018

JeffRausch commented Aug 23, 2018

stevejroberts commented Aug 23, 2018

stevejroberts commented Aug 24, 2018

JeffRausch commented Aug 24, 2018 •

edited

stevejroberts commented Aug 27, 2018

stevejroberts commented Aug 27, 2018

stevejroberts commented Aug 29, 2018

JeffRausch commented Aug 30, 2018

CloudFormation with ChangeSet - Task fails if there are no changes #97

CloudFormation with ChangeSet - Task fails if there are no changes #97

Comments

JeffRausch commented Aug 22, 2018

stevejroberts commented Aug 22, 2018 • edited

stevejroberts commented Aug 22, 2018

JeffRausch commented Aug 22, 2018

stevejroberts commented Aug 22, 2018

JeffRausch commented Aug 23, 2018

stevejroberts commented Aug 23, 2018

stevejroberts commented Aug 24, 2018

JeffRausch commented Aug 24, 2018 • edited

stevejroberts commented Aug 27, 2018

stevejroberts commented Aug 27, 2018

stevejroberts commented Aug 29, 2018

JeffRausch commented Aug 30, 2018

stevejroberts commented Aug 22, 2018 •

edited

JeffRausch commented Aug 24, 2018 •

edited