Terraform forcing replacement of ECS task definition - (only on task definitions with mount points)

[im-lcoupe]

Summary

Hi there,

So this only seems to have become a problem since upgrading my code to the latest version - and strangely only seems to happen on the task definitions with mount points (however, the format of them hasn't changed...)

Terraform will constantly try and replace the two task definitions regardless of whether any changes have been made to them...

Any guidance on this would be greatly appreciated, as it means the task definition revision is changing on every run (which of course is not ideal)...

Terraform Version

0.12.18

Terraform Configuration Files - 1st problematic task definition - followed by the second

[
  {
    "name": "${container_name_nginx}",
    "image": "${container_image_nginx}",
    "memory": ${container_memory},
    "cpu": ${container_cpu},
    "networkMode": "awsvpc",
    "volumesFrom": [],
    "essential": true,
    "portMappings": [
      {
        "containerPort": 443,
        "hostPort": 443,
        "protocol": "tcp"
      }
    ],
    "mountPoints": [
     {
    "readOnly": false,
    "containerPath": "/var/www/symfony/var/log",
    "sourceVolume": "shared_symfony_logs"
     },
     {
     "readOnly": false,
     "containerPath": "/var/log/nginx",
     "sourceVolume": "shared_nginx_logs"
     }
    ],
    "environment": [
      {
        "name": "${env_var_1_name}",
        "value": "${env_var_value_1}"
      },
      {
        "name": "${env_var_2_name}",
        "value": "${env_var_value_2}"
      },
      {
        "name": "${env_var_3_name}",
        "value": "${env_var_value_3}"
      }
    ],
  "logConfiguration" : {
    "logDriver" : "awslogs",
    "options" :{
      "awslogs-create-group": "true",
      "awslogs-group": "${container_name_nginx}",
      "awslogs-region": "${platform_region}",
      "awslogs-stream-prefix": "ecs"
    }
  }
},

{
  "name": "${container_name_php}",
  "image": "${container_image_php}",
  "memory": ${container_memory},
  "cpu": ${container_cpu},
  "networkMode": "awsvpc",
  "volumesFrom": [],
  "essential": true,
  "portMappings": [
    {
      "containerPort": 9000,
      "hostPort": 9000,
      "protocol": "tcp"
    }
  ],
  "mountPoints": [
   {
  "readOnly": false,
  "containerPath": "/var/www/symfony/var/log",
  "sourceVolume": "shared_symfony_logs"
   },
   {
   "readOnly": false,
   "containerPath": "/var/log/nginx",
   "sourceVolume": "shared_nginx_logs"
   }
  ],
  "environment": [
    {
      "name": "${env_var_4_name}",
      "value": "${env_var_value_4}"
    },
    {
      "name": "${env_var_5_name}",
      "value": "${env_var_value_5}"
    },
    {
      "name": "${env_var_6_name}",
      "value": "${env_var_value_6}"
    },
    {
      "name": "${env_var_7_name}",
      "value": "${env_var_value_7}"
    },
    {
      "name": "${env_var_8_name}",
      "value": "${env_var_value_8}"
    },
    {
      "name": "${env_var_10_name}",
      "value": "${env_var_value_10}"
    },
    {
      "name": "${env_var_11_name}",
      "value": "${env_var_value_11}"
    },
    {
      "name": "${env_var_12_name}",
      "value": "${env_var_value_12}"
    },
    {
      "name": "${env_var_13_name}",
      "value": "${env_var_value_13}"
    },
    {
      "name": "${env_var_14_name}",
      "value": "${env_var_value_14}"
    },
    {
      "name": "${env_var_15_name}",
      "value": "${env_var_value_15}"
    },
    {
      "name": "${env_var_16_name}",
      "value": "${env_var_value_16}"
    },
    {
      "name": "${env_var_17_name}",
      "value": "${env_var_value_17}"
    },
    {
      "name": "${env_var_18_name}",
      "value": "${env_var_value_18}"
    },
    {
      "name": "${env_var_19_name}",
      "value": "${env_var_value_19}"
    },
    {
      "name": "${env_var_20_name}",
      "value": "${env_var_value_20}"
    },
    {
      "name": "${env_var_21_name}",
      "value": "${env_var_value_21}"
    },
    {
      "name": "${env_var_22_name}",
      "value": "${env_var_value_22}"
    },
    {
      "name": "${env_var_1_name}",
      "value": "${env_var_value_1}"
    },
    {
      "name": "${env_var_28_name}",
      "value": "${env_var_value_28}"
    },
    {
      "name": "${env_var_29_name}",
      "value": "${env_var_value_29}"
    },
    {
      "name": "${env_var_30_name}",
      "value": "${env_var_value_30}"
    },
    {
      "name": "${env_var_31_name}",
      "value": "${env_var_value_31}"
    },
    {
      "name": "${env_var_32_name}",
      "value": "${env_var_value_32}"
    },
    {
      "name": "${env_var_33_name}",
      "value": "${env_var_value_33}"
    },
    {
      "name": "${env_var_34_name}",
      "value": "${env_var_value_34}"
    },
    {
      "name": "${env_var_35_name}",
      "value": "${env_var_value_35}"
    },
    {
      "name": "${env_var_36_name}",
      "value": "${env_var_value_36}"
    },
    {
      "name": "${env_var_37_name}",
      "value": "${env_var_value_37}"
    },
    {
      "name": "${env_var_38_name}",
      "value": "${env_var_value_38}"
    }
  ],
  "secrets":[
    {
      "name":"${sensitive_var_1}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_1}"
    },
    {
      "name":"${sensitive_var_2}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_2}"
    },
    {
      "name":"${sensitive_var_3}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_3}"
    },
    {
      "name":"${sensitive_var_4}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_4}"
    },
    {
      "name":"${sensitive_var_5}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_5}"
    },
    {
      "name":"${sensitive_var_6}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_6}"
    },
    {
      "name":"${sensitive_var_7}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_7}"
    },
    {
      "name":"${sensitive_var_8}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_8}"
    },
    {
      "name":"${sensitive_var_9}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_9}"
    },
    {
      "name":"${sensitive_var_10}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_10}"
    },
    {
      "name":"${sensitive_var_11}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_11}"
    },
    {
      "name":"${sensitive_var_12}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_12}"
    },
    {
      "name":"${sensitive_var_13}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_13}"
    }
  ],
"logConfiguration" : {
  "logDriver" : "awslogs",
  "options" :{
    "awslogs-create-group": "true",
    "awslogs-group": "${container_name_php}",
    "awslogs-region": "${platform_region}",
    "awslogs-stream-prefix": "ecs"
  }
}
},


{
  "name": "${container_name_logstash}",
  "image": "${container_image_logstash}",
  "memory": ${container_memory},
  "cpu": ${container_cpu},
  "networkMode": "awsvpc",
  "volumesFrom": [],
  "essential": true,
  "portMappings": [
    {
      "containerPort": 9600,
      "hostPort": 9600,
      "protocol": "tcp"
    }
  ],
  "mountPoints": [
   {
  "readOnly": false,
  "containerPath": "/var/www/symfony/var/log",
  "sourceVolume": "shared_symfony_logs"
   },
   {
   "readOnly": false,
   "containerPath": "/var/log/nginx",
   "sourceVolume": "shared_nginx_logs"
   }
 ],
 "environment": [
   {
     "name": "${env_var_23_name}",
     "value": "${env_var_value_23}"
   },
   {
     "name": "${env_var_1_name}",
     "value": "${env_var_value_1}"
   }
 ],
"logConfiguration" : {
  "logDriver" : "awslogs",
  "options" :{
    "awslogs-create-group": "true",
    "awslogs-group": "${container_name_logstash}",
    "awslogs-region": "${platform_region}",
    "awslogs-stream-prefix": "ecs"
    }
  }
}
]

Second Task definition

[
{
  "name": "${container_name_php}",
  "image": "${container_image_php}",
  "memory": ${container_memory},
  "cpu": ${container_cpu},
  "networkMode": "awsvpc",
  "volumesFrom": [],
  "essential": true,
  "command": ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"],
  "portMappings": [
    {
      "containerPort": 9001,
      "hostPort": 9001,
      "protocol": "tcp"
    }
  ],
  "mountPoints": [
   {
  "readOnly": false,
  "containerPath": "/var/www/symfony/var/log",
  "sourceVolume": "shared_worker_logs"
   }
 ],
  "environment": [
    {
      "name": "${env_var_4_name}",
      "value": "${env_var_value_4}"
    },
    {
      "name": "${env_var_5_name}",
      "value": "${env_var_value_5}"
    },
    {
      "name": "${env_var_6_name}",
      "value": "${env_var_value_6}"
    },
    {
      "name": "${env_var_7_name}",
      "value": "${env_var_value_7}"
    },
    {
      "name": "${env_var_8_name}",
      "value": "${env_var_value_8}"
    },
    {
      "name": "${env_var_10_name}",
      "value": "${env_var_value_10}"
    },
    {
      "name": "${env_var_11_name}",
      "value": "${env_var_value_11}"
    },
    {
      "name": "${env_var_12_name}",
      "value": "${env_var_value_12}"
    },
    {
      "name": "${env_var_13_name}",
      "value": "${env_var_value_13}"
    },
    {
      "name": "${env_var_14_name}",
      "value": "${env_var_value_14}"
    },
    {
      "name": "${env_var_15_name}",
      "value": "${env_var_value_15}"
    },
    {
      "name": "${env_var_16_name}",
      "value": "${env_var_value_16}"
    },
    {
      "name": "${env_var_17_name}",
      "value": "${env_var_value_17}"
    },
    {
      "name": "${env_var_18_name}",
      "value": "${env_var_value_18}"
    },
    {
      "name": "${env_var_19_name}",
      "value": "${env_var_value_19}"
    },
    {
      "name": "${env_var_20_name}",
      "value": "${env_var_value_20}"
    },
    {
      "name": "${env_var_21_name}",
      "value": "${env_var_value_21}"
    },
    {
      "name": "${env_var_22_name}",
      "value": "${env_var_value_22}"
    },
    {
      "name": "${env_var_1_name}",
      "value": "${env_var_value_1}"
    },
    {
      "name": "${env_var_28_name}",
      "value": "${env_var_value_28}"
    },
    {
      "name": "${env_var_29_name}",
      "value": "${env_var_value_29}"
    },
    {
      "name": "${env_var_30_name}",
      "value": "${env_var_value_30}"
    },
    {
      "name": "${env_var_31_name}",
      "value": "${env_var_value_31}"
    },
    {
      "name": "${env_var_32_name}",
      "value": "${env_var_value_32}"
    },
    {
      "name": "${env_var_33_name}",
      "value": "${env_var_value_33}"
    },
    {
      "name": "${env_var_34_name}",
      "value": "${env_var_value_34}"
    },
    {
      "name": "${env_var_35_name}",
      "value": "${env_var_value_35}"
    },
    {
      "name": "${env_var_36_name}",
      "value": "${env_var_value_37}"
    },
    {
      "name": "${env_var_37_name}",
      "value": "${env_var_value_37}"
    },
    {
      "name": "${env_var_38_name}",
      "value": "${env_var_value_38}"
    }
  ],
  "secrets":[
    {
      "name":"${sensitive_var_1}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_1}"
    },
    {
      "name":"${sensitive_var_2}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_2}"
    },
    {
      "name":"${sensitive_var_3}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_3}"
    },
    {
      "name":"${sensitive_var_4}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_4}"
    },
    {
      "name":"${sensitive_var_5}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_5}"
    },
    {
      "name":"${sensitive_var_6}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_6}"
    },
    {
      "name":"${sensitive_var_7}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_7}"
    },
    {
      "name":"${sensitive_var_8}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_8}"
    },
    {
      "name":"${sensitive_var_9}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_9}"
    },
    {
      "name":"${sensitive_var_10}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_10}"
    },
    {
      "name":"${sensitive_var_11}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_11}"
    },
    {
      "name":"${sensitive_var_12}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_12}"
    },
    {
      "name":"${sensitive_var_13}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_13}"
    }
  ],
"logConfiguration" : {
  "logDriver" : "awslogs",
  "options" :{
    "awslogs-create-group": "true",
    "awslogs-group": "${container_name_php}",
    "awslogs-region": "${platform_region}",
    "awslogs-stream-prefix": "ecs"
  }
}
},


{
  "name": "${container_name_logstash}",
  "image": "${container_image_logstash}",
  "memory": ${container_memory},
  "cpu": ${container_cpu},
  "networkMode": "awsvpc",
  "volumesFrom": [],
  "essential": true,
  "portMappings": [
    {
      "containerPort": 9600,
      "hostPort": 9600,
      "protocol": "tcp"
    }
  ],
  "mountPoints": [
   {
  "readOnly": false,
  "containerPath": "/var/www/symfony/var/log",
  "sourceVolume": "shared_worker_logs"
   }
 ],
 "environment": [
   {
     "name": "${env_var_23_name}",
     "value": "${env_var_value_23}"
   },
   {
     "name": "${env_var_1_name}",
     "value": "${env_var_value_1}"
   }
 ],
"logConfiguration" : {
  "logDriver" : "awslogs",
  "options" :{
    "awslogs-create-group": "true",
    "awslogs-group": "${container_name_logstash}",
    "awslogs-region": "${platform_region}",
    "awslogs-stream-prefix": "ecs"
    }
  }
}
]

Example plan output - to me it isn't clear what exactly it needs to change which requires a forced replacement - it looks to remove vars that are already there and then re add them? in other places it seems to re order them also. It also looks to be adding the network mode (awsvpc) which is already defined in the task definition?

      ~ container_definitions    = jsonencode(
          ~ [ # forces replacement
              ~ {
                    cpu              = 512
                  ~ environment      = [
                      - {
                          - name  = "PHP_FPM_PORT"
                          - value = "9000"
                        },
                      - {
                          - name  = "PHP_FPM_HOST"
                          - value = "localhost"
                        },
                        {
                            name  = "APP_ENV"
                            value = "prod"
                        },
                      + {
                          + name  = "PHP_FPM_HOST"
                          + value = "localhost"
                        },
                      + {
                          + name  = "PHP_FPM_PORT"
                          + value = "9000"
                        },
                    ]
                    essential        = true
                    image            = "###########################:latest"
                    logConfiguration = {
                        logDriver = "awslogs"
                        options   = {
                            awslogs-create-group  = "true"
                            awslogs-group         = "###############"
                            awslogs-region        = "eu-west-1"
                            awslogs-stream-prefix = "ecs"
                        }
                    }
                    memory           = 1024
                    mountPoints      = [
                        {
                            containerPath = "/var/www/symfony/var/log"
                            readOnly      = false
                            sourceVolume  = "shared_symfony_logs"
                        },
                        {
                            containerPath = "/var/log/nginx"
                            readOnly      = false
                            sourceVolume  = "shared_nginx_logs"
                        },
                    ]
                    name             = "##################"
                  + networkMode      = "awsvpc"
                    portMappings     = [
                        {
                            containerPort = 443
                            hostPort      = 443
                            protocol      = "tcp"
                        },
                    ]
                    volumesFrom      = []
                } # forces replacement,

Expected Behavior

Terraform should not try and replace the task definitions on every plan.

Actual Behavior

Terraform forces replacement of the task definition on every plan.

Steps to Reproduce

Terraform plan

[ghost]

This issue has been automatically migrated to hashicorp/terraform-provider-aws#11526 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to hashicorp/terraform-provider-aws#11526.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.