aws_ecs: "Resource timed out waiting for completion" error during stack deletion #25969
Labels
@aws-cdk/aws-ecs
Related to Amazon Elastic Container
bug
This issue is a bug.
effort/medium
Medium work item – several days of effort
investigating
This issue is being investigated and/or work is in progress to resolve the issue.
p2
Describe the bug
I am running the ECS cluster with ASG as capacity provider (due to GPU loads) on one single EC2.
In order to avoid app down times during ecs task update I have set enable_managed_scaling=True in ecs.AsgCapacityProvider() with the goal that ecs first spins up a new instance, places task on it and only after that the previous instance will be deregistered and terminated.
Enabling of managed scaling adds two CloudWatch alarms behind the scenes.
The problem is now that the instance termination happens now only after 15 minutes according to the alarm setting.
During stack deletion I obtain "Resource timed out waiting for completion" error, which crashes the CI/CD pipeline, which manages the stacks.
I have not found way to override the 15 min setting on the template, since this is how it looks in it.
Expected Behavior
Enabling of managed scaling in the ECS for ASG capacity provider either does not have "collisions" with stack timeouts or there is a way to alter the CloudWatch rules (e.g. lower the 15 min threshold) via cdk.
Current Behavior
During stack deletion with enable_managed_scaling=True in ecs.AsgCapacityProvider() "Resource timed out waiting for completion" error will be raised and stack deletion fails
Reproduction Steps
In order to reproduce the issue a lot of components must be deployed so I can assist with further information if needed since the stack is up and running
Possible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.83.0
Framework Version
No response
Node.js Version
18
OS
Ubuntu 20.04 LTS
Language
Python
Language Version
3.1.0.6
Other information
No response
The text was updated successfully, but these errors were encountered: