Delete Clusters with Active Services during ECS Cleanup #1799
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
** What ** 1. Add logic to scale down ecs cluster services before deleting them to avoid getting a 400 on deletion of active services 2. Fix describeTasks invocation to include task list 3. Fix buggy expiry time logic when checking for tasks to delete ** Why ** This allows the ecs cleanup script to handle deletion of clusters whose services have failed tasks, or have tasks that have been open for more than a week.
agarakan
force-pushed
the
cleanup_ecs_active_services
branch
from
268f45f to
3060eaa
Compare
agarakan
commented
Jul 30, 2025
| } | ||
| } | ||
|
|
||
| func isClusterTasksExpired(ctx context.Context, client *ecs.Client, clusterArn *string) bool { |
There was a problem hiding this comment.
Previously this logic was failing with an incorrect request, where describeTaskInput was missing the Tasks parameter. This now retrieves the tasks and then corrects the describeTask call
agarakan
commented
Jul 30, 2025
| continue | ||
| } | ||
|
|
||
| for _, service := range services.ServiceArns { |
There was a problem hiding this comment.
Now handles deleting clusters with active services by performing Service ScaleDown and then deletion. Validated the original 400 is no longer encountered. See 400 in PR description
agarakan
commented
Jul 30, 2025
| // Clean ECS clusters if they have been running longer than 7 days | ||
|
|
||
| var expirationTimeOneWeek = time.Now().UTC().Add(clean.KeepDurationOneWeek) | ||
| var expirationTimeOneWeek = time.Now().UTC().Add(-clean.KeepDurationOneWeek) |
There was a problem hiding this comment.
Fixed bug. Expiration time used to be set 1 week in the future.
agarakan
commented
Jul 30, 2025
| if !strings.HasPrefix(*cluster.ClusterName, "cwagent-integ-test-cluster-") { | ||
| continue | ||
| } | ||
| if cluster.ActiveServicesCount > 0 { |
There was a problem hiding this comment.
Check not needed since we handle activeServiceCount in deletion now
TravisStark
previously approved these changes
Jul 30, 2025
agarakan
force-pushed
the
cleanup_ecs_active_services
branch
from
0138137 to
3060eaa
Compare
TravisStark
approved these changes
Jul 30, 2025
Paramadon
approved these changes
Jul 30, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of the issue
This allows the ecs cleanup script to handle deletion of clusters whose services have failed tasks, or have tasks that have been open for more than a week.
See old output of buggy ECS Resource Cleanup run (clean-ecs-clusters): https://github.com/aws/amazon-cloudwatch-agent/actions/runs/16610452973/job/46992332357
Description of changes
Ex:
License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
Ran locally with developer account
See fix in kicked-off resource cleanup in github runner (see clean-ecs-clusters): https://github.com/aws/amazon-cloudwatch-agent/actions/runs/16631933178/job/47063290921
Requirements
Before commiting your code, please do the following steps.
make fmtandmake fmt-shmake lintIntegration Tests
To run integration tests against this PR, add the
ready for testinglabel.