(hotswap): Hotswap always deploys all resources changed since last full deployment

[comcalvi]

Running cdk deploy --hotswap will update all hotswapped resources since the last CloudFormation deployment.

Suppose a stack contains two lambda functions, lambda1 and lambda2. Then a change to lambda1 is deployed with --hotswap. Now a change to lambda2 is made. When these changes to lambda2 are deployed with --hotswap, the changes to lambda1 are also deployed, even though they were deployed by a previous hotswap.

This happens because all potentially hotswappable changes are found by diffing the current app template with the template in CloudFormation, which is not updated by hotswaps. Thus subsequent --hotswap deployments do not know about previous --hotswap deployments; that is, hotswaps are not aware of drift introduced by previous hotswaps.

This could incur a performance penalty over time. The time it takes for a --hotswap deployment to complete is proportional to the number of changed resources, not the number of changes. If many resources are changed over the course of many --hotswap deployments, then the performance will degrade. If many changes are made to only a few resources, the performance change is negligible.

To avoid this performance loss, perform a regular deployment. This will remove all drift, meaning that the next hotswap will not suffer the performance penalty from the previously accumulated drift.

Edit by @mrgrain

#888 poses a related problem. When reverting back to the original template after a change that was hotswapped, hotswap does not recognise this as a change anymore and therefore will not run.

[skinny85]

There is also an interesting functional correctness problem caused by this issue, in addition to the performance problem - the inability to rollback to the last CloudFormation state. Let me illustrate with an example.

Let's say you have a Lambda function in your Stack lambda1, and you perform a full cdk deployment (no --hotswap) for it with its code asset at version A. Now, you change the code of the Lambda, let's call the new code version B, and perform a hotswap deployment (cdk deply --hotswap). The code of the running function gets updated to B.

However, you decided that you don't like the new changes, and you want to get back to the code at version A. You roll back the changes to the Lambda code locally, and run cdk deploy --hotswap. However, this command exits saying "no changes", and the Lambda code is still at version B! What happened?

Well, the CloudFormation diff that cdk deploy --hotswap is based on actually claims there are no changes, because the last full CloudFormation deployment was for version A, which is also the version currently in the CDK code!

This is actually a tricky situation to get out of, because not even a full deployment (cdk deploy, no --hotswap) has effect here (as it will also detect "no changes").

Because of this problem, we might have to tackle this issue sooner than if it was only a performance problem.

[p0wl]

If using --hotswap during local development, the =ask flag (suggested here: https://github.com/aws/aws-cdk-rfcs/blob/master/text/0001-cdk-update.md) could actually help to prevent this situation.

I just reread the description of =ask and now I'm surprised that it asks on Stack level, I would have expected it ask me for every lambda function. So that if I update lambda1 and cdk diff also detects a change in lambda2, I can tell via cli to only hot swap lambda1.

What do you think of that? Is the =ask flag already work in progress?

[skinny85]

We haven't started on the =ask flag yet @p0wl, but this is a very interesting idea 🙂.