aws_ecs_task_definition and continuous delivery to ecs

[hashibot]

This issue was originally opened by @dennari as hashicorp/terraform#13005. It was migrated here as part of the provider split. The original body of the issue is below.

---

With the task and container definition data sources I'm almost able to get our continuous delivery setup to play nicely with Terraform. We rebuild the docker image with a unique tag at every deployment. This means that after the CI service redeploys a service, the corresponding task definition's revision is incremented and the image field in a container definition changes.

I dont' seem to be able to create a setup where the task definition could be managed by Terraform in this scenario.

Terraform Version

v0.9.1

Affected Resource(s)

• resource aws_ecs_task_definition

Terraform Configuration Files

# Simply specify the family to find the latest ACTIVE revision in that family.
data "aws_ecs_task_definition" "mongo" {
  task_definition = "${aws_ecs_task_definition.mongo.family}"
}
data "aws_ecs_container_definition" "mongo" {
  task_definition = "${data.aws_ecs_task_definition.mongo.id}"
  container_name  = "mongodb"
}

resource "aws_ecs_cluster" "foo" {
  name = "foo"
}

resource "aws_ecs_task_definition" "mongo" {
  family = "mongodb"
  container_definitions = <<DEFINITION
[
  {
    "cpu": 128,
    "environment": [{
      "name": "SECRET",
      "value": "KEY"
    }],
    "essential": true,
    "image": "${aws_ecs_container_definition.mongo.image}",
    "memory": 128,
    "memoryReservation": 64,
    "name": "mongodb"
  }
]
DEFINITION
}

resource "aws_ecs_service" "mongo" {
  name          = "mongo"
  cluster       = "${aws_ecs_cluster.foo.id}"
  desired_count = 2
  # Track the latest ACTIVE revision
  task_definition = "${aws_ecs_task_definition.mongo.family}:${max("${aws_ecs_task_definition.mongo.revision}", "${data.aws_ecs_task_definition.mongo.revision}")}"
}

The problem is then that after a CI deployment, terraform would like to create a new task definition. The task definition resource here points to an earlier revision and the image field is considered changed.

With the deprecated template resources, I was able to ignore changes to variables which solved this issue. One solution that comes to mind would be the ability to set revision of the aws_ecs_task_definition resource.

I'd be grateful for any and all insights.

[kurtwheeler]

I have run into this issue as well. I think the solution I am going to go with is to not have the task definition be managed by terraform. Circle CI has a blog post about how to push a new task definition via a script they provide.

I agree that the ability to set the aws_ecs_task_definition would enable managing the task definition via Terraform. However philosophically it does seem to break Terraform's model of having resource blocks correspond to resources within AWS. If that parameter were added then a single aws_ecs_task_definition resource block would be responsible for creating multiple AWS resources.

[JDiPierro]

I've gotten around this by using terraform taint as part of the deploy process:

• Push the container to the ECR
• terraform taint the task-def
• terraform apply makes a new revision and updates the service

[naveenb29]

@JDiPierro While using the taint solution - does it not kill the current task and replace rather than deploy the new task and then drain the old one ?

[JDiPierro]

@naveenb29 Nope. I believe that would be the case if you were tainting the ECS Service. Since just the task-def is being recreated the ECS service is updated causing the new tasks to deploy. ECS waits for them to become healthy and then kills the old containers.

[tomelliff]

Our CI process is tagging the image as it's pushed to ECR and then passing that tag to the task definition. This automatically leads to the task definition changing so that Terraform knows to recreate it and then that's linked to the ECS service causing it to update the service with the new task definition.

I think I'm missing the issue that others are having here.

That said, I'd like for Terraform to (optionally) wait for the deployment to complete: new task definition has running tasks equal to the desired count and, potentially, that the other task definitions have been deregistered so no new traffic will reach them. I can't see a nice way to get at that information using the API though as the events don't really expose enough information so you'd probably have to describe the old task def, find the running tasks using it, find the ports they were running on, check the ALB they are registered to has all of those ports set as draining.

For now I'm simply shelling out and waiting until the PRIMARY service deployment has a running count equal to the desired count (doesn't catch the short period from PRIMARY tasks being registered to old tasks being deregistered) or waiting until the deployment list has a length of 1 (all old task definitions have been completely drained which is overkill as new connections won't arrive there so the deployment can be considered completed before this).

[radeksimko]

Hi folks,
see more detailed explanation of why this is happening at #13 (comment)

Labelling appropriately...

[dmikalova]

I was able to solve the inactive task definition issue with the example in the ECS task definition data source. You set up the ECS service resource to use the the max revision of either what your Terraform resource has created, or what is in the AWS console which the data source retrieves.

The one downside to this is if someone changes the task definition, Terraform will not realign that to what's defined in code.

[Esya]

What do you guys think about having a remote backend (It's S3 in my case), and having your CI pipeline create the new task definition and changing the .tfstate file directly to match it ?

For example, mine looks like this :

"aws_ecs_task_definition.backend_service": {
    "type": "aws_ecs_task_definition",
    "depends_on": [
        "data.template_file.task_definition"
    ],
    "primary": {
        "id": "backend-service",
        "attributes": {
            "arn": "arn:aws:ecs:eu-west-1:REDACTED:task-definition/backend-service:8", // This could be changed manually
            "container_definitions": "REDACTED",
            "family": "backend-service",
            "id": "backend-service",
            "network_mode": "",
            "placement_constraints.#": "0",
            "revision": "8", // This could be increased manually
            "task_role_arn": ""
        },
        "meta": {},
        "tainted": false
    },
    "deposed": [],
    "provider": ""
},

Couldn't we just change the arn and the revision, so that the next time terraform runs, it still thinks it has the "latest version" of the task defintion in it's state ?

[chriswhelix]

I'm not sure I understand the problem y'all are trying to solve here. Why not just use terraform to create the new task definition in the first place, and then your tf state is always consistent? Our setup is similar to what @tomelliff describes.

[Esya]

@chriswhelix Well in my particular case, I have two separate repositories. One that holds the terraform project, and it creates my ECS cluster, my services, and the initial task definition.

The other one is for a specific service, and I'd like to have some CI/continuous delivery flow in place (Using gitlab pipelines in my case) to "containerize" the project, push it to ECR, and trigger a service update on my ECS cluster. (Edit: as a reminder, currently, if we use the aws cli to do this as part of our CI workflow, then the next terraform run will overwrite the task def.)

So, when you say "use terraform to create the new task definition in the first place", are you implying that on our CI system, when pushing our service's code, we should also clone our terraform repo, change the variable that holds the image tag or that service, do a terraform apply, and commit + push to the TF repository?

tl;dr: Need a way to trigger service updates from any of our projects' build pipeline, without any user interaction with terraform.

[chriswhelix]

@Esya what we do is that each project has in its build config the version of the terraform repo it is expecting to be deployed with. When the CI pipeline is ready to deploy, it pulls down the terraform repo using the git tag specified in the project build config, then runs terraform against that, providing the image tag it just wrote to the ECR repo as an input variable.

We don't write down the ECR image tag in the terraform repo; it must be provided each time terraform is run. So that avoids simple code updates to projects requiring any change to the terraform repo.

[schmod]

I'm using ecs-deploy in my deployment pipeline, and a terraform config that looks something like this:

# Gets the CURRENT task definition from AWS, reflecting anything that's been deployed
# outside of Terraform (ie. CI builds).
data "aws_ecs_task_definition" "task" {
  task_definition = "${aws_ecs_task_definition.main.family}"
}

# "Dummy" application for initial deployment
data "aws_ecr_repository" "sample" {
  name = "sample"
}

# ECR Repo for our actual app
data "aws_ecr_repository" "main" {
  name = "${var.ecr_name}"
}

resource "aws_ecs_task_definition" "main" {
  family = "${var.name}"
  task_role_arn = "${module.iam_roles.ecs_service_deployment_role_arn}"

  container_definitions = <<DEFINITION
[
  {
    "name": "${var.name}",
    "image": "${data.aws_ecr_repository.sample.repository_url}:latest",
    "essential": true,
    "portMappings": [{
      "containerPort": ${var.container_port},
      "hostPort": 0
    }]
  }
]
DEFINITION
}

resource "aws_ecs_service" "main" {
  name = "${var.name}"
  cluster = "${var.cluster}"
  desired_count = 2
  task_definition = "${aws_ecs_task_definition.main.family}:${max("${aws_ecs_task_definition.main.revision}", "${data.aws_ecs_task_definition.task.revision}")}"
  iam_role = "${module.iam_roles.ecs_service_deployment_role_arn}"

}

During the initial deployment, Terraform deploys an "empty" container. When the CI pipeline runs, ecs-deploy creates a new task definition revision with the newly-built image/tag, and updates the service accordingly.

Terraform recognizes these new deployments via data.aws_ecs_task_definition.task, and doesn't attempt to overwrite them.

HOWEVER, if other parts of the task definition change, Terraform will redeploy the sample application, as it'll try to create a new revision of the task definition (using the config containing the sample application). Hypothetically, data.aws_ecs_container_definition could be used to pull the image of the currently-active task definition. However, I haven't been able to figure out a way to use this that doesn't create a circular dependency or result in a chicken/egg problem during the initial deployment (ie. the data source is looking for a task definition that hasn't been created yet):

data "aws_ecs_task_definition" "task" {
  task_definition = "${aws_ecs_task_definition.main.family}"
}

data "aws_ecs_container_definition" "task" {
  task_definition = "${data.aws_ecs_task_definition.task.id}"
  container_name  = "${var.name}"
}

resource "aws_ecs_task_definition" "main" {
  family = "${var.name}"
  task_role_arn = "${module.iam_roles.ecs_service_deployment_role_arn}"

  container_definitions = <<DEFINITION
[
  {
    "name": "${var.name}",
    "image": "${data.aws_ecs_container_definition.task.image}",
  }
]
DEFINITION
}

This creates a cycle, and won't work during the initial deployment.

This is very close to my ideal setup. If Terraform somehow supported a "get or create" data/resource hybrid, I'd be able to do almost exactly what I'm looking for.