Nested Stack Change Set Evaluation

[atkinsonm]

Duplicate of https://forums.aws.amazon.com/thread.jspa?start=0&threadID=228489&tstart=0, posted Mar 30, 2016.

The current behavior of evaluating a change set on the "root" stack is that all nested stacks will show "conditional" changes regardless if the underlying templates at the TemplateURLs have changed.

This enhancement request is for recursive change set evaluation for nested stacks. Ideally, a change set evaluated on the "root" stack should show all additions/replacements/deletions/changes for all child stacks recursively. It should also automatically resolve any inter-stack dependencies and evaluate those as part of the changes.

[ghost]

Agreed, without recursive change set the risk of unintended change is significantly higher just like the old days before change set was a thing. we need that visibility.

[bjorg]

I second that. I've been thinking about computing change-sets manually myself for sub-stacks, but it's a major pain I wish AWS could solve for me. It feels rather undifferentiated cruft, but super important for reducing risks to updates.

[PatMyron]

See also #224 about a new UpdateReplacePolicy option so every change set Replacement value no longer needs to be evaluated

[VilleBez]

See also this thread Change set support for nested stacks? at Mar 30, 2016

[philipakash]

Any pointers on when this will be released or at least move to Work in Progress?

[rjlohan]

This is a bigger work item than a typical service coverage ask so needs a bit more investment in design and engineering. It's being looked at, but I can't offer any more specific timeline than that at this time.

[ghost]

This is a bigger work item than a typical service coverage ask so needs a bit more investment in design and engineering. It's being looked at, but I can't offer any more specific timeline than that at this time.

Thanks Dude, we know its a big task and appreciate that you guys are looking at investing in this, its an important feature to safe usage of nested stacks.

[Patrick-56Bit]

Is there an update on this?

All my stacks are "modularized" using nested stacks. Unfortunately, I have reached a point where it is scary updating a root stack as I'm not 100% sure what will happen in the child stacks. This is a critical feature in my opinion.

[moby04]

@rjlohan We had 4th anniversary 2 days ago since this issue was reported on aws forum (https://forums.aws.amazon.com/thread.jspa?threadID=228489&start=0&tstart=0) so please don't state the obvious like "it's bigger work item". Instead, could you please provide any information when (or rather if) this feature can happen?

[Patrick-56Bit]

@rjlohan We had 4th anniversary 2 days ago since this issue was reported on aws forum (https://forums.aws.amazon.com/thread.jspa?threadID=228489&start=0&tstart=0) so please don't state the obvious like "it's bigger work item". Instead, could you please provide any information when (or rather if) this feature can happen?

@rjlohan Can you at least tell us if this is possible and whether there is the will to do it. If not I will be looking elsewhere re IaC on AWS, but at least it gives us a better position to make a decision. We need to know whether we should give up on the numerous modules we wrote using CloudFormation and rewrite them using something else, or whether to wait just a bit more. Its been 4 years, the least we deserve is a straight answer even if it doesn't include a timeline.

[clayvan]

+1 On this being an important feature.

CloudFormation best practices are to use nested stacks, and when making an update, to do that update to the parent stack. However the parent stack doesn't provide visibility into the changes being made. It seems like those are conflicting best practices to me.

I'm at the point where I'm considering getting rid of nested stacks completely and moving to a model of 1 massive CloudFormation file.

[clayvan]

Update on that theory, there is a limit of 200 resources on a given CF stack, so my model of moving to 1 massive CF file is not even possible. Now I seem to be at a crossroads and I have 2 options forward.

1. I move away from CF altogether and onto Terraform which is a bummer for me. I like terraform for a lot of reasons, but specific to this issue I like the fact that terraform shows me exactly what is going to change when making an update. I don't like the fact that we are heavily invested in the AWS ecosystem, and something as small as this issue is the dealbreaker.

2. I have to create some sort of changeset automation. Essentially running changesets against my nested stacks and pass those changes back up into a central location for me to view. I don't want to have to create a workaround like this. Especially considering it seems like this is exactly what CF should be doing in the first place. All the tools and information are there for me to create this automation, so why can CF not natively do this?

I understand it's a bigger work effort, but like I outlined above, it seems like all the tools and information are already there, so how hard could this really be to implement?

[bjorg]

I thought about 2 as well. However, if the nested stacks use attributes from other resources, you would not be able to determine if they get updated or not until the resources have updated.

For example, let's assume the nested stack needs a resource ARN from its parent stack. In a subsequent update, a property is changed on that resource. If the property can be updated in-place, then nested stack has no changes, but if the property requires replacement of the resource, then the nested stack needs to update as well.

[philipakash]

Implementing 2 is equal to "better to go to another solution". I am still under the hope that CF will someday announce the changeset feature availability.
Downsides of this I have recommended people to not take this solution as its risky to rely on a CI/CD model that deploys nested stacks. The value of the CF just goes down because of this problem.
I think even if this feature is complex to implement, any small indicator around the changeset would do that would keep the customers using CF.

[moby04]

@bjorg
Yes, however you request update of master stack. This means you are able to calculate master's replacements before calculating parameters used by nested ones. That comes down to proper recursive calculation - I admit that it wouldn't be trivial one but definitely possible. And it is useful/necessary exactly because it is not a trivial task.

[benbridts]

For those thinking about ways to work around this, in some cases you can also work with layered / paralel stacks. This article does a pretty good job explaining how that works. I personally haven't had a stack that couldn't be split up this way (although I know the number of resources can climb quickly for lambda + api-gw stacks):

https://blog.rowanudell.com/cloudformation-layer-cake-pattern/

[bjorg]

Actually, even without knowing what changes (i.e. update vs replace) might occur, it would still be beneficial to do a structural comparison on the nested stack. That said, if it contains conditional logic, it may be difficult to determine what is triggered without knowing the parameter values.

[rjlohan]

@Patrick-56Bit Yes, we believe it is possible; yes, there is enough will to do it. This remains a top customer ask. I've just got an update from the team assigned to do this work. Transparently; we may be able to show progress on this early next year, but we’re being particularly mindful of operational implications and backwards compatibility challenges we’ve found as we’ve done deep research on this recently.

[mliner]

@rjlohan so that means early next year, that's it? We use cloudformation to handle dozens of accounts/applications (with use of nested stacks). However we are still experiencing to have cloudformation API kinda "half-baked" and features are being released in a hurry without full support. We have to "fix" lot of them with custom macros. If I have to wait another year for such crucial functionality like changesets for nested stacks, it is maybe feasible to migrate hundreds of templates into terraform. I was hoping that AWS can show the same pace for fixing issues with current services as bringing new ones.

[Patrick-56Bit]

@mliner I agree with you that AWS is way too slow to fix and improve this compared to launching new services. We are slowly migrating away from Cloudformation because of this. But we found it easier to use the AWS CDK, which is a high-level API to Cloudformation, then going towards a completely different system like Terraform (which btw is also fantastic). We are still in the beginning of this process so may revert our decision and go to Terraform, but till now CDK is working great.

[babeal]

I'm plus 1 on this as well. I need to be able to see all changes related to the change set in child stacks. Layered cake we had before and it was no good. Takes too long to deploy new stacks, you need to remember order, and heaven forbid you use global exports for sharing variables. Our layered cake can't even be removed because it was removed in the wrong order and now resources have to be recreated by hand to be able to complete the removal. It's so bad we are just going to destroy the account.

[babeal]

@Patrick-56Bit with the CDK, how did that solve the change set problem?

[bjorg]

I don't see how CDK helps here. It's also CloudFormation.

IHMO, the problem is not really nested stacks, which are inherently impossible to predict (b/c they might use output attributes from custom resources, for example). Alternatively, you can think of nested stacks as custom resources. We don't see what's going on in the inside them either. However, if CloudFormation resource limits were higher than 200, then we wouldn't need nested stacks for many scenarios in the first place.

[babeal]

@bjorg yea, its possible that there is a change in a passed input parameter to a child stack that has an affect in a conditional that may be difficult to determine. However in other cases where it is deterministic if we could get the list of resources and what may happen to them so that the risk can be evaluated I would be happy. I actually use child stacks in a way that's very helpful to me. For example one child stack will create a task definition specific to a micro service and output references that can be used by a universal child stack that defines scaling policies. This way my devs can choose which type of scaling policy they want by just changing the template url. I do the same thing for api gateway depending if its vpc linked into the private subnet or if i want it public (dev vs prod) deployments. Now if i was using CDK, then you're absolutely correct that one big file would work because the abstractions would be at the code (typescript) and I wouldn't be too concerned about the overall structure. This is where we are eventually headed, unless Terraform is that much better. It seems that terraform has better support for things like calculations and variables which cloudformation is severely lacking. Right now the only way to do something like that is to use a lambda which is quite painful for something as simple as toLower()

[bjorg]

@babeal what you're saying makes absolutely sense and a good use-case. I'm not debating any of it. I have made my peace with the notion that nested stacks and custom resources are opaque abstractions. I actually think they are the same thing: both take parameters, both have outputs. Mathematically speaking, they are identical as there are nothing to distinguish them from each other.

On the topic of toLower() don't get me started!!! :) There seems so much low-hanging fruit with adding some basic intrinsic functions, like Fn::StartsWith, which I had to emulate the hard way!

[Patrick-56Bit]

@bjorg @babeal CDK is still very new and a work in progress but we found the abstraction a breath of fresh air. Since it makes use of "normal" languages (we use Typescript) we can use toLower(), for loops, if..then..else, etc without issue. That is fantastic and makes it even better than Terraform in that regard.

There are slight differences in how to develop the applications that one should be aware of though, that we learned the hard way. For example coming from a CloudFormation background we used cloudformation parameters to pass context but that didn't work. We learnt that CDK had its own "context" parameters that are available separately from Cloudformation and those worked great.

As for changeset detection in child stacks when using nested stacks, it is not perfect but the abstraction layer (as well as the fact it is open source) allows the community to iterate and improve this cloudformation issue, at a faster rate. Case in point this pull request looks like it will partially show differences inside nested stacks too. It is not approved yet, but hopefully will be soon. Using the cdk diff command of the cdk cli one should see most (not all due to the nature of nested stacks) changes. Faster innovation and the ability to use common programming features are all a step in the right direction.

One problem with CDK (and obviously cloudformation) that will probably bite us in the near future is the fact that they are AWS specific, with the industry moving towards multi-cloud. Terraform isn't vendor specific but then you don't have the same capabilities of a fully-fledged language. Pulumi would be the only multi-vendor, abstracted IaC engine I have heard of, but haven't used it myself.

[PatMyron]

Change sets for nested stacks documenation

[jrd-nonlilly]

@PatMyron Is there corresponding documentation for the API somewhere? That new --include-nested-stacks CLI parameter has a minor typo on that page; the first time it's introduced, the two dashes appear to have been collapsed into an em- or en-dash.

[luiseduardocolon]

@jrd-nonlilly Hey John, API docs are here: https://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_CreateChangeSet.html
and yes, I saw the typo on the CLI page too, but it should not affect your use.
👍

[jrd-nonlilly]

@luiseduardocolon sure, just wanted to point it out.

Thank you for the link. We're quite happy to see this roll out.

[RichardHuntCCI]

Great to see this roll out!

We use CodePipeline to perform CloudFormation actions (CreateChangeSet, ExecuteChangeSet), and I notice the nested stack change evaluation doesn't seem to be applying by default here.

Is there any plan to add the "include-nested-stacks" evaluation as a configuration property for the CloudFormation action?

[herbertmuraro]

While using --include-nested-stacks option, when parameters of a nested stack resource refers an output from another nested stack resource using Ref! or Fn::GetAtt it does not recurse the change detection and worst it shows incorrect information on the nested stack. It'd be better not showing anything instead of informing the replacement of some resources.

In the below example, although there're no changes to the bastion stack, it shows that there're changes and resources will be replaced.

Also, when some nested resource uses Fn::Select(1, ...), it fails the change set creation. I suppose it's because the mappings are created during execution and it's not possible to evaluate them, but in case there're no changes in the inputs it shouldn't mark it for change at all.

.. with that, this feature becomes useless unless there are predefined parameters on the nested stacks.

The same changeset creates fine when --include-nested-stacks option is disabled.

[wjr1985]

I'm seeing the same behavior as @herbertmuraro with respect to the nested change sets indicating modifications that don't exist when there's a !Ref or Fn::GetAtt that refers to another nested stack - in my case it seems to happen either when it's passed in as Parameter, or when it's retrieved from the value of a nested stack inside the current nested stack

[bjorg]

I'm speculating here, but it sounds like change propagations cannot be tracked across stack boundaries. When that happens, it's safest to assume a value changes rather than staying the same. Consequently, it may appear resources are being replaced when in fact they are not. This makes change sets show false positives, which are a problem for pipelines that alert on these types of situations.

Question is, can this behavior be fixed without being a breaking change? Or are we locked in for the long term now?

[luiseduardocolon]

Closing since this was released :)

[herbertmuraro]

Ok, it is closed now, but I have the feeling that the issues are still there.

[dehli]

I'm also seeing that behavior. Went ahead and created a new issue with the context you've provided since this one has been closed (#759).

[wizofaus]

How do we use --include-nested-stacks when using aws cloudformation deploy?