ECS task definition not deregistered

[Sebasti1492]

Jenkins and plugins versions report

Environment

Jenkins non production

What Operating System are you using (both controller, and any agents involved in the problem)?

ECS Fargate agent launched using the plugin - amazon-ecs-plugin

Reproduction steps

1. Install the Amazon Elastic Container Service (ECS) / Fargate plugin in Jenkins.
2. Login to Jenkins. Manage Jenkins -> Manage Nodes and Clouds -> Configure Clouds
3. Add new EC2/Elastic Container service cloud.
4. Configure the ECS cluster and role details and the agent template with label for eg: aloc-linux-agent
5. Run a test job pointing to restrict run to this agent.
6. Adhoc job starts, ECS task definition created, Task started and job completes.
7. Trigger the same job, the task definition is again created new.
8. The ECS task definitions in console are getting piled up with new version every time and not deregistered.

Expected Results

Once a new ECS task definition is created and registered, the old ECS task definition (previous version) should be deregistered.

Actual Results

The ECS task definitions in console are getting piled up with new version every time a job runs and not deregistered.

Anything else?

No response

[morfien101]

I think what you are referring to is actually a problem in ECS itself and not this plugin.

see: aws/containers-roadmap#685

It is not possible to delete task definitions. They go to inactive rather.

I see the same issue in my setup. The task definitions are piling up as I use more dynamically created agents.

[Sebasti1492]

@morfien101
In my case all the task definitions are active and it piles up for every single job for an agent created through the plugin and I have 5+ agents running approx 100+ jobs every day and eventually end up 1k+ active task definitions in a week or two.

As per below in the plugin code, is the plugin not supposed to deregister it?
https://github.com/jenkinsci/amazon-ecs-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSService.java

/**
     * Deregisters a task definition created for a template we are deleting.
     * It's expected that taskDefinitionArn is set
     * We don't attempt to de-register anything if TaskDefinitionOverride isn't null
     *
     * @param template       The template used to create the task definition
     * @return The task definition if found, otherwise null
     */
    void removeTemplate(final ECSTaskTemplate template) {
        AmazonECS client = clientSupplier.get();

        //no task definition was created for this template to delete
        if (template.getTaskDefinitionOverride() != null) {
            return;
        }

        String taskDefinitionArn = template.getDynamicTaskDefinition();
        try {
            if (taskDefinitionArn != null) {
                client.deregisterTaskDefinition(
                        new DeregisterTaskDefinitionRequest().withTaskDefinition(taskDefinitionArn));
            }

        } catch (ClientException e) {
            LOGGER.log(Level.WARNING, "Error de-registering task definition: " + taskDefinitionArn, e);
        }
    }

[morfien101]

yea, I seen your comment in another issue. I feel you and can easily see how that would be annoying.

[Stericson]

@Sebasti1492 Can you clarify whether or not you're using Task Definition Override? As mentioned here, these aren't deregistered if you're using that.

[bencehornak-gls]

In the meantime AWS has implemented aws/containers-roadmap#685, so de-registering task definitions after terminating the workers would be a really cool feature! (we've got >100k dangling task definitions, which are just too many for our cloud security scanner tools)