Skip to content

svc delete hangs at UPDATE_COMPLETE_CLEANUP_IN_PROGRESS #1882

@iamhopaul123

Description

@iamhopaul123

I ran a copilot one-off task in an environment and the env stack hung at UPDATE_COMPLETE_CLEANUP_IN_PROGRESS status when i ran copilot svc delete to delete the last ALB service in that environment. The CFN events said it was because it failed to delete the ALB security group because it has a dependent object.

Under the hood it is because when you run a task in an app env cluster, copilot will find all the security groups tagged with app env tags, and pass those security groups to ecs:TaskRun. However, since it adds both environment security group and ALB security group, it creates an ENI and attaches both env security group and ALB security group to the ENI. As a result, when doing svc delete and the env controller is trying to delete the ALB security group, it fails because the security group is still attaching to the ENI. In this situation you have to stop the running one-off task in order to remove the ENI before deleting the service.

Given the reason above, since for now we don't allow importing security groups, we should only use the environment security group when we run a task in an environment.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/taskIssues about taskstype/bugIssues that are bugs.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions