integ-runner: Integration tests succeed, yet the integ-runner throws errors on "describeStackEvents" calls #27445
Comments
diranged
added
bug
|
Update The actual cause for the test failure (the |
indrora
added
the
response-requested
|
This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled. |
github-actions
bot
added
the
closing-soon
peterwoodworth
removed
closing-soon
|
I'm getting the same errors when deploying nested stack. It deploys successfully but has those errors. StackNames are defined everywhere. |
pahud
added
p2
effort/medium
|
@pahud Still occurring with 2.102.0...it happens with any nested stack on an incredibly simple project. |
|
I should note, it only happens on initial stack creation... we don't see the error on subsequent updates. |
|
Seeing the same issue. Also with a very simple project. |
|
I have been getting this issue in all my simple CDK projects, but somehow, it seems to still work. error: Here is a simple example. I even trimmed it down, and still get this (only on first deploy) const cluster = new eks.Cluster(this, 'jc-cdk-eks-xp-04-cluster', {
clusterName: "jc-cdk-eks-xp-04-cluster",
version: eks.KubernetesVersion.V1_27,
defaultCapacity: 1,
defaultCapacityInstance: new ec2.InstanceType('t3.micro'),
kubectlLayer: new KubectlV27Layer(this, 'kubectl'),
});I even specified the const app = new cdk.App();
new JcCdkEksXp04Stack(app, 'JcCdkEksXp04Stack', {
stackName: "JcCdkEksXp04Stack"
}); |
|
Was able to replicate this in 2.102.0. Here's a sample code: Got the following error: |
|
Still happens on 2.103.1 |
|
We are also having this issue since we upgraded to 2.103.1 |
|
I'm getting same errors on a similar scenario. :( I use also the CDK version 2.103.1 here some info:
|
|
I am not getting the error anymore with CDK 2.104.0 |
|
i downgraded to 2.99.1 and it worked again. |
|
without digging in that much, i'd imagine it might have something to do with this? |
ahh.. Key word "not". It looks like it might be fixed with #27692 |
|
Issue persists for me with 2.104.0 |
|
with the CDK 2.104.0 is for me again ok!. no errors by deployment |
|
Closing, as this has been resolved. I was surprised to notice that the CDK team didn't close this. |
|
Describe the bug
We recently upgraded from
2.93.0to2.100.0... once we did, we saw a strange behavior with one of our integration tests on a large stack with nested stacks. The launching of the test stacks work fine (there are no errors in the CREATE or DELETE process), but during the tests we get errors printed out to the console that eventually cause the test-case itself to fail:The test creation phase ...
Then there's the deletion phase:
[20:25:36] Notices refreshed [20:25:37] CDK toolkit version: 2.100.0 (build e1b5c77) [20:25:37] Command line arguments: { _: [ 'deploy' ], rollback: false, 'outputs-file': 'cdk-integ.out.integ.xx-cluster.ts.snapshot/assertion-results.json', O: 'cdk-integ.out.integ.xx-cluster.ts.snapshot/assertion-results.json', outputsFile: 'cdk-integ.out.integ.xx-cluster.ts.snapshot/assertion-results.json', 'require-approval': 'never', ... test/integ/constructs/aws-eks/cdk-integ.out.integ.xx-cluster.ts.snapshot/bundling-temp-8d8dd6365075d75b5065745e14ac81d8b26551cbab856be8fee4b434bd30d711/index.js 1.3mb ⚠️ ✨ Synthesis time: 37.49s [20:26:14] Checking for previously published assets ,... IntegTest/DefaultTest/DeployAssert: destroying... [1/2] [20:26:22] Retrieved account ID 12345678910 from disk cache [20:26:22] Assuming role 'arn:aws:iam::12345678910:role/cdk-hnb659fds-deploy-role-12345678910-us-east-2'. [20:26:22] Call failed: describeStacks({"StackName":"IntegTestDefaultTestDeployAssertE3E7D2A4"}) => Stack with id IntegTestDefaultTestDeployAssertE3E7D2A4 does not exist (code=ValidationError) ✅ IntegTest/DefaultTest/DeployAssert: destroyed INFRA-CDK-K8S-ClusterTest: destroying... [2/2] [20:26:22] Retrieved account ID 12345678910 from disk cache [20:26:23] Waiting for stack INFRA-CDK-K8S-ClusterTest to finish creating or updating... [20:26:23] Stack INFRA-CDK-K8S-ClusterTest has an ongoing operation in progress and is not stable (DELETE_IN_PROGRESS (User Initiated)) INFRA-CDK-K8S-ClusterTest-SupplementalPluginsNestedStackSupplementalPluginsNestedStack-1Y25214IX1XHB | 0 | 8:26:25 PM | DELETE_IN_PROGRESS | AWS::CloudFormation::Stack | INFRA-CDK-K8S-ClusterTest-SupplementalPluginsNestedStackSupplementalPluginsNestedStack-1Y25214IX1XHB User Initiated INFRA-CDK-K8S-ClusterTest-SupplementalPluginsNestedStackSupplementalPluginsNestedStack-1Y25214IX1XHB | 0 | 8:26:28 PM | DELETE_IN_PROGRESS | Custom::AWSCDK-EKS-KubernetesResource | SupplementalPlugins/ExternalSecrets/ServiceAccount/manifest-ServiceAccountServiceAccountResource/Resource/Default (ExternalSecretsServiceAccountmanifestServiceAccountServiceAccountResource54655F82) ... operation in progress and is not stable (DELETE_IN_PROGRESS) [20:52:06] Call failed: describeStacks({"StackName":"INFRA-CDK-K8S-ClusterTest"}) => Stack with id INFRA-CDK-K8S-ClusterTest does not exist (code=ValidationError) [20:52:06] Stack INFRA-CDK-K8S-ClusterTest does not exist [20:52:06] Call failed: describeStackEvents({"StackName":"INFRA-CDK-K8S-ClusterTest"}) => Stack [INFRA-CDK-K8S-ClusterTest] does not exist (code=ValidationError) [20:52:06] Call failed: describeStackEvents({"StackName":"INFRA-CDK-K8S-ClusterTest"}) => Stack [INFRA-CDK-K8S-ClusterTest] does not exist (code=ValidationError) ✅ INFRA-CDK-K8S-ClusterTest: destroyed [20:52:06] Reading cached notices from /Users/myuser/.cdk/cache/notices.json FAILED integ/constructs/aws-eks/integ.xx-cluster-IntegTest/DefaultTest (undefined/us-east-2) 3058.427s Integration test failed: TypeError [ERR_STREAM_NULL_VALUES]: May not write null values to stream Test Results: Tests: 1 failed, 1 total --- Integration test metrics --- Profile undefined + Region us-east-2 total time: 3058.428 /Users/myuser/git/myorg/infra-cdk-k8s/test/integ/constructs/aws-eks/integ.xx-cluster.ts: 3058.428 Error: Some integration tests failed! at main (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/@aws-cdk/integ-runner/lib/index.js:10401:15) 👾 Task "integ" failed when executing "npx integ-runner --update-on-failed -vv test/integ/constructs/aws-eks/integ.xx-cluster.ts -vvvvv --parallel-regions=us-east-2" (cwd: /Users/myuser/git/myorg/infra-cdk-k8s) error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. $ echo $? 1Expected Behavior
I do not expect to see errors thrown during the integration tests - especially when the stacks themselves actually work and our deployments work.
Current Behavior
We're seeing a strange error outputted about the
stackNamebeing invalid during some validation call... but yet the tests continue, and the stack itself seems fine.Reproduction Steps
I don't really know right now... we have multiple integration tests, and we only get this error on our largest test... this test takes 45m+ to complete (eks clusters + vpcs), and the codebase is very large so it's not something I can easily share. I can say that the test has one stack with many nested stacks.
Possible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.100.0
Framework Version
No response
Node.js Version
18
OS
OSX and Linux
Language
TypeScript
Language Version
No response
Other information
I'm rolling back to our last known good release (2.93.0)... then I will roll forward one release at a time, but since the tests take nearly an hour to do each.. it will take a while before I can track down which release specifically introduces the bug.
The text was updated successfully, but these errors were encountered: