integ tests on private cdk lib fail since two days - ideas how to debug welcome

[markussiebert]

Since two days we see the integ tests failing in our ci pipelines. It looks like:

👾 release » build » test » eslint | eslint --ext .ts,.tsx --fix --no-error-on-unmatched-pattern  src test build-tools projenrc .projenrc.ts
👾 release » build » test » integ:EksAddonClusterAutoscaler:assert | [ -d "test/EksAddons/EksAddonClusterAutoscaler.integ.snapshot" ] || (echo "No snapshot available for integration test 'EksAddonClusterAutoscaler'. Run 'projen integ:EksAddonClusterAutoscaler:deploy' to capture." && exit 1)
👾 release » build » test » integ:EksAddonClusterAutoscaler:assert | cdk synth --app "ts-node -P tsconfig.dev.json test/EksAddons/EksAddonClusterAutoscaler.integ.ts" --no-notices --no-version-reporting --no-asset-metadata --no-path-metadata -o test/EksAddons/.tmp/EksAddonClusterAutoscaler.integ/assert.cdk.out > /dev/null
👾 release » build » test » integ:EksAddonClusterAutoscaler:assert | diff -r -x asset.* -x cdk.out -x manifest.json -x tree.json test/EksAddons/EksAddonClusterAutoscaler.integ.snapshot/ test/EksAddons/.tmp/EksAddonClusterAutoscaler.integ/assert.cdk.out/
diff: test/EksAddons/.tmp/EksAddonClusterAutoscaler.integ/assert.cdk.out/: No such file or directory
👾 Task "release » build » test » integ:EksAddonClusterAutoscaler:assert" failed when executing "diff -r -x asset.* -x cdk.out -x manifest.json -x tree.json test/EksAddons/EksAddonClusterAutoscaler.integ.snapshot/ test/EksAddons/.tmp/EksAddonClusterAutoscaler.integ/assert.cdk.out/" (cwd: /builds/a/cdk/eks)
error Command failed with exit code 1.

The thing I don't understand is, why is this happening - even pipelines that already were successfull fail now on rerunning them. The day things started to fail had no coincedence with changes in our ci environment - or codechanges - as mentioned, we see this failure even on builds that already succeeded.

Locally everything works fine - no errors.

Do you have an idea how to debug this, or what can cause this behaviour?

[mrgrain]

With no changes in your codebase? 🤯

To me the likeliest reasons seems that this command stops producing the correct output and fails silently:

cdk synth --app "ts-node -P tsconfig.dev.json test/EksAddons/EksAddonClusterAutoscaler.integ.ts" --no-notices --no-version-reporting --no-asset-metadata --no-path-metadata -o test/EksAddons/.tmp/EksAddonClusterAutoscaler.integ/assert.cdk.out > /dev/null

When you run this locally, can you make sure the .tmp folder is removed first?

[markussiebert]

Yes - after digging some time in the ci container it seems cdk is not working at all - the command finishes with exit code 0 even if the integ.test.ts is deleted ... if I change it to cdk deploy, it's finished in no time with no output and exit code 0. of course no cdk.out is created

[mrgrain]

Very odd! I would see that the app command does without the cdk cli. Sometimes that's giving a better error message. (That's the ts-node...)

[mrgrain]

@markussiebert Did you ever get to the bottom of this?

[markussiebert]

No was debugging in the gitlab executor pod - finally switched node image and it worked agin. Hadn't pinned the char, so maybe there was a change in the node image we used then...

[mrgrain]

Okay. Closing as this seems to be an issue with Node or the AWS CDK.