CDK deploy stopped updating playbook.zip asset

[cartalla]

Describe the bug

I use an ansible playbook to configure my ec2 instances. I create an s3 asset for the playbook directory which gets put into a zip file. I then download the playbook in my userdata, unzip it, and run it. Until the last day, when I'd update any file in the playbook and deployed the stack, the asset would get updated which would cause a userdata change which would cause the instance to reboot and run the new playbook contents (I have userdata configured to rerun on reboot).

Now, I'm not seeing the userdata change and it keeps downloading the old asset. This has been working for over a year, so why has this suddenly stopped working? This is a complex CDK and I'm not quite sure how to debug this. It seems like core CDK functionality has just ceased to work.

Expected Behavior

New cdk asset gets uploaded to S3 and the asset reference in the userdata changes, instance gets stoppped/started, userdata runs and gets the latest asset.

Current Behavior

Asset doesn't change. Ansible runs with old playbook contents.

Reproduction Steps

Change file in asset directory.
Update stack.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.69.0

Framework Version

No response

Node.js Version

16.15.0

OS

Amazon Linux 2

Language

Python

Language Version

3.7.16

Other information

No response

[pahud]

Thanks for your report. Is there anything we can do to reproduce this in our environment?

[cartalla]

I'm not sure. I'm doing some debug now. When I look in the cdk.out directory I see the new asset get created with the correct contents. This is reflected in slurmeda_assets.json and slurmeda.template.json. When I update the stack then it says no updates, but the userdata on the instance doesn't match the userdata in slurmeda.template.json. So maybe the problem is with the diff?

[cartalla]

I'm an AWS employee, so you can slack me directly if that's more convenient.

[cartalla]

Might be a CFN problem, not CDK. I looked at the cdk.out and compared the asset in the deployed CFN template and they matched. But the userdata on the instance itself was still referring to an old asset. The userdata script on the instance didn't match the deployed template.

[cartalla]

I think a configuration change on the instance was preventing its metadata script from being update.

Not a cdk problem.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.