Allow empty updates

[justinberry]

Hey everyone,

We're using the update-stack[-cs] command in our build pipeline to update our stacks based on repository changes, and if there are no changes in our cloud formation templates (e.g. maybe we refactored a yaml file that has no effect on the template) sceptre attempts to issue an update and returns a non-zero status and subsequently fails the build. Right now I'm parsing the StatusReason to find if the update actually failed or just lacked any new changes, but would like something a bit nicer.

Some ideas for handling this:

1. Return a zero status message for these commands when there are no changes to be applied.
2. Add a flag that allows updates to contain no changes and 'fail' silently.
3. Supply a status reason that's more CLI friend than what StatusReason provides. Right now it's much more human-friendly and not the greatest thing to rely on if they someday decide to change the wording.

I'm new to this tool so apologies if I've missed something obvious here.

[theseanything]

Is it just throwing the no changes boto exception? We should probably handle that exception for the create_change_set part and return a zero status.

[justinberry]

In a somewhat related issue, I notice that if there are any other types of errors in updating via update-stack-cs it will automatically initiate a rollback then also return 0. For example, I set our desired capacity for our ASG to be 1 when the minimum was 3. This throws the error UPDATE_FAILED Desired capacity:1 must be between the specified min size:3 and max size:10, then rolls back.

The problem is detecting errors from the CLI end, as the command still returns a 0 after encountering an error. Perhaps more broadly, defining a strategy of detecting and handling errors on the command line would be beneficial.

[justinberry]

So I've had a chance to poke around and do some testing with the latest code - it doesn't appear that an exception is thrown from boto, instead the change set is created, fails on the aws end because there are no changes to apply (as expected), then upon checking if status != StackChangeSetStatus.READY (or StackStatus.COMPLETE in the case where change sets are not used) it immediately exits with status 1 in the update_command.

By tracing through one of these updates, I found that cloudformation returns:

Status: FAILED
ExecutionStatus: UNAVAILABLE
StatusReason: The submitted information didn't contain changes. Submit different information to create a change set.

Which results in stack._get_cs_status returning a StackChangeSetStatus.DEFUNCT status, leading to the non-zero exit directive being called.

From what I can tell, it seems the only way to detect this scenario is to check that the status reason matches the above snippet, and return 0. However I'm unsure if this is a change you would agree with. I'm happy to attempt a PR if you do, however.

[ngfgrant]

@justinberry would this be specifically for the CLI output. I don't see any reason why we couldn't include a more meaningful CLI output for this so that you could add some logic to the returned message on your side. We probably wouldn't want to start creating our own exit codes for responses from boto but could certainly help with more useful responses for you to parse?

[mithun]

I was able to work-around this issue by using a dummy resource. See the recommendation in this forum post.

Adding a resource of type AWS::CloudFormation::WaitConditionHandle to your template will always ensure that an update-stack[-cs] is always run since the dummy resource will always indicate a change.

As a bonus, this also works very well when you are changing outputs and other metadata which does not always show up as an update.

[theseanything]

Do we want exit codes to reflect on just how Sceptre runs or also reflect on CloudFormation?

[justinberry]

In my view this sort of ties into the philosophy of the tool - is it supposed to build a layer of abstraction on top of boto/CF and add some features that people are missing? Or does it lean more towards being an easy tool for a developer/sysadmin to use while sometimes ignoring how the underlying implementations of boto and CF work? Or something else?

If it's the latter, I'd argue that if there's a good argument for overruling an error that improves usability it may be worth doing. In this case, it seems to me that amazon are treating a what amounts to a NOOP action as an error, and the state you're sending up (i.e. the template) is actually valid. Also given exit codes are pretty central to error handling on the command line, it is a key part of making the tool usable in automated environments. Perhaps there are reasons that you wouldn't want this, but I can't think of any scenarios where that'd be the case right now.

Another possible solution would be to add something like a failure_hook, where the boto exception/stack status is passed to a user-defined hook, and then people can do with it what they like, whether it be returning exit code 42 or sending an email to their grandmother. This wouldn't break any of what you have, but would allow complete control over however a developer/sysadmin may want to handle their errors.

@ngfgrant - friendly/parseable output would work, however it just means we'd still need to add stuff around the sceptre invocation to handle the output (piping it to jq or whatever), increasing the complexity of using the tool. Still viable though if you'd prefer to not go down the path of my other suggestions.

@mithun - this is a pretty cool solution however it does add a little more noise to our templates, but it may come in handy for us, thanks for the link!

[zaro0508]

sceptre-date-resolver has been implemented in issue #608 which may help with this issue.