Container Definition error when deploying an app and fluentbit container

[sfiguereo]

Description

The module gives and error when deploying container definition with variables and secrets for two containers, and app and a fluentbit container.
Error: Invalid for_each argument
on .terraform/modules/account_microservice.app_ecs/modules/service/main.tf line 531, in module "container_definition":
for_each = { for k, v in var.container_definitions : k => v if local.create_task_definition && try(v.create, true) }
local.create_task_definition is true
var.container_definitions will be known only after apply
The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.

When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.

Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.

When managing the task and container definitions outside the module, the problem is:

Error: Provider produced inconsistent final plan
When expanding the plan for module.account_service.aws_ecs_task_definition.app to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .container_definitions: was cty.StringVal(""), but now cty.StringVal("[{"Command":["/bin/sh","-c","/home/node/entrypoint.sh"],"Cpu":1024,"CredentialSpecs":null,"DependsOn":null,"DisableNetworking":null,"DnsSearchDomains":null,"DnsServers":null,"DockerLabels":null,"DockerSecurityOptions":null,"EntryPoint":null,"Environment":[{"Name":"APP_NAME","Value":"account"},{"Name":"AWS_ORGANIZATION_ARN","Value":"arn:aws:organizations::137452182988:organization/o-exlib790lh"},{"Name":"AWS_ORGANIZATION_MASTER_ACCOUNT_ID","Value":"137452182988"},{"Name":"AWS_REGION","Value":"us-east-1"},{"Name":"BRANCH_IO_URL","Value":"https://api2.branch.io/v1/url"},{"Name":"DATABASE_HOST","Value":"accountdb.cmba1foxdib9.us-east-1.rds.amazonaws.com"},{"Name":"DATABASE_NAME","Value":"accountdb"},{"Name":"DATABASE_PORT","Value":"5432"},{"Name":"DATABASE_SCHEMA","Value":"account"},{"Name":"DATABASE_USER","Value":"stagewoodAdmin"},{"Name":"FIREBASE_DATABASE_URL","Value":"https://dtyket-5b4f1-default-rtdb.firebaseio.com"},{"Name":"FIREBASE_PROJECT_ID","Value":"dtyket-5b4f1"},{"Name":"FIREHOSE_NAME","Value":"ctrldev-DeliveryStream"},{"Name":"FIREHOSE_ROLE_ARN_TO_ASSUME","Value":"arn:aws:iam::905418284744:role/FirehoseRole"},{"Name":"PORT","Value":"3010"},{"Name":"RABBITMQ_USERNAME","Value":"stagewoodAdmin"}],"EnvironmentFiles":null,"Essential":true,"ExtraHosts":null,"FirelensConfiguration":null,"HealthCheck":null,"Hostname":null,"Image":"591947861756.dkr.ecr.us-east-1.amazonaws.com/ssodev-account-app:latest","Interactive":false,"Links":null,"LinuxParameters":null,"LogConfiguration":null,"Memory":8192,"MemoryReservation":null,"MountPoints":null,"Name":"account","PortMappings":null,"Privileged":false,"PseudoTerminal":false,"ReadonlyRootFilesystem":null,"RepositoryCredentials":null,"ResourceRequirements":null,"Secrets":[{"Name":"BRANCH_KEY","ValueFrom":"arn:aws:ssm:us-east-1:591947861756:parameter/ecs/account/branch_key"},{"Name":"DATABASE_PASSWORD","ValueFrom":"arn:aws:secretsmanager:us-east-1:591947861756:secret:rds!db-d8d4bc55-bc6a-4160-aa84-d787178c1924-O0cbWt"},{"Name":"FIREBASE_CLIENT_EMAIL","ValueFrom":"arn:aws:ssm:us-east-1:591947861756:parameter/ecs/account/firebase_client_email"},{"Name":"FIREBASE_PRIVATE_KEY","ValueFrom":"arn:aws:ssm:us-east-1:591947861756:parameter/ecs/account/firebase_private_key"},{"Name":"RABBITMQ_PASSWORD","ValueFrom":"arn:aws:secretsmanager:us-east-1:905418284744:secret:central-broker/stagewoodAdmin_password-AiixaC"}],"StartTimeout":30,"StopTimeout":120,"SystemControls":null,"Ulimits":null,"User":"0","VolumesFrom":null,"WorkingDirectory":null},{"Command":null,"Cpu":512,"CredentialSpecs":null,"DependsOn":null,"DisableNetworking":null,"DnsSearchDomains":null,"DnsServers":null,"DockerLabels":null,"DockerSecurityOptions":null,"EntryPoint":null,"Environment":[{"Name":"APP_NAME","Value":"account"},{"Name":"AWS_REGION","Value":"us-east-1"},{"Name":"FIREHOSE_NAME","Value":"ctrldev-DeliveryStream"},{"Name":"FIREHOSE_ROLE_ARN_TO_ASSUME","Value":"arn:aws:iam::905418284744:role/FirehoseRole"}],"EnvironmentFiles":null,"Essential":true,"ExtraHosts":null,"FirelensConfiguration":null,"HealthCheck":null,"Hostname":null,"Image":"176238661673.dkr.ecr.us-east-1.amazonaws.com/fluent-bit:latest","Interactive":false,"Links":null,"LinuxParameters":null,"LogConfiguration":null,"Memory":1024,"MemoryReservation":null,"MountPoints":null,"Name":"fluent-bit","PortMappings":null,"Privileged":false,"PseudoTerminal":false,"ReadonlyRootFilesystem":null,"RepositoryCredentials":null,"ResourceRequirements":null,"Secrets":null,"StartTimeout":30,"StopTimeout":120,"SystemControls":null,"Ulimits":null,"User":"0","VolumesFrom":null,"WorkingDirectory":null}]").

This is a bug in the provider, which should be reported in the provider's own issue tracker.

• [ yes] ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]: latest

• Terraform version: 1.9.2

• Provider version(s):

Reproduction Code [Required]

Code for first Error with full ECS Module use:
module "app_ecs" {
source = "terraform-aws-modules/ecs/aws"

cluster_name = "${var.naming_prefix}-${local.container_name}-ecs_cluster"

create_cloudwatch_log_group = true
cluster_configuration = {
execute_command_configuration = {
logging = "OVERRIDE"
log_configuration = {
create_log_group = true
cloud_watch_log_group_name = "/aws/ecs/cluster/${local.container_name}"
retention_in_days = 30
}
}
}

Capacity providers

fargate_capacity_providers = {
FARGATE = {
default_capacity_provider_strategy = {
weight = 50
base = 20
}
}
FARGATE_SPOT = {
default_capacity_provider_strategy = {
weight = 50
}
}
}

services = {

(local.container_name) = {
  subnet_ids = module.vpc.private_subnets
  cpu        = 2048
  memory     = 16384
  runtimePlatform = {
    cpuArchitecture       = "X86_64"
    operatingSystemFamily = "LINUX"
  }

  load_balancer = {
    service = {
      target_group_arn = module.alb_app.target_groups["app_tg"].arn
      container_name   = local.container_name
      container_port   = local.container_port
    },
    fluent_bit_service = {
      target_group_arn = module.alb_app.target_groups["fluent_bit_tg"].arn
      container_name   = "fluent-bit"
      container_port   = 2020
    }
  }

  service_registries = {
    provider = "aws.central_caar"
    registry_arn   = var.service_discovery_arn
    container_name = "${local.container_name}"
    container_port = local.service_discovery_port
  }

  security_group_rules = {
    alb_ingress = {
      type                     = "ingress"
      from_port                = local.container_port
      to_port                  = local.container_port
      protocol                 = "tcp"
      description              = "Service port"
      source_security_group_id = module.alb_sg.security_group_id
    }
    service_discovery_ingress = {
      type                     = "ingress"
      from_port                = local.service_discovery_port
      to_port                  = local.service_discovery_port
      description              = "Service Discovery Port"
      protocol                 = "tcp"
      source_security_group_id = module.alb_sg.security_group_id
    }
    fluent_bit_http_server = {
      type                     = "ingress"
      from_port                = 2020
      to_port                  = 2020
      protocol                 = "tcp"
      description              = "Fluent Bit HTTP server port"
      source_security_group_id = module.alb_sg.security_group_id
    }
    fluent_bit_access = {
      type        = "ingress"
      from_port   = 24224
      to_port     = 24224
      protocol    = "tcp"
      description = "Fluent Bit internal opearations server port"
      cidr_blocks = ["${var.vpc_cidr_block}"]
    }
    egress_all = {
      type        = "egress"
      from_port   = 0
      to_port     = 0
      protocol    = "-1"
      cidr_blocks = ["0.0.0.0/0"]
    }
  }

  ingress_with_cidr_blocks = [
    {
      description = "Custom HTTP traffic for Fluent Bit on port 2020"
      from_port   = 2020
      to_port     = 2020
      protocol    = "tcp"
      cidr_blocks = var.vpc_cidr_block
    }
  ]

  # Service IAM roles references (actual roles defined in resources below)
  create_iam_role              = false
  iam_role_arn                 = aws_iam_role.ecs_task_execution_role.arn
  iam_role_name                = aws_iam_role.ecs_task_execution_role.name
  iam_role_unique_id           = aws_iam_role.ecs_task_execution_role.unique_id
  create_task_exec_iam_role    = false
  task_exec_iam_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  task_exec_iam_role_name      = aws_iam_role.ecs_task_execution_role.name
  task_exec_iam_role_unique_id = aws_iam_role.ecs_task_execution_role.unique_id
  create_task_iam_role         = false
  task_iam_role_arn            = aws_iam_role.ecs_task_role.arn
  task_iam_role_name           = aws_iam_role.ecs_task_role.name
  task_iam_role_unique_id      = aws_iam_role.ecs_task_role.unique_id
  create_tasks_iam_role        = false
  tasks_iam_role_arn           = aws_iam_role.ecs_task_role.arn
  tasks_iam_role_name          = aws_iam_role.ecs_task_role.name
  tasks_iam_role_unique_id     = aws_iam_role.ecs_task_role.unique_id

  ##############################################################################
  # CONTAINERS

  container_definitions = {

    ############################################################################
    # APPLICATION CONTAINER

    (local.container_name) = {
      image              = "${module.ecr_app.repository_url}:latest"
      cpu                = 1024
      memory             = 8192
      memory_reservation = 2048
      port_mappings = [
        {
          name          = local.container_name
          containerPort = local.container_port
          hostPort      = local.container_port
          protocol      = "tcp"
          appProtocol   = "http"
        },
        {
          name          = "${local.container_name}-${var.env}-service"
          containerPort = local.service_discovery_port
          hostPort      = local.service_discovery_port
          protocol      = "tcp"
          appProtocol   = "http"
        },
      ]
      essential = true
      command = [
        "/bin/sh",
        "-c",
        "/home/node/entrypoint.sh"
      ],
      environment = [
        {
          name  = "APP_NAME"
          value = "${local.container_name}"
        },
        {
          name  = "AWS_ORGANIZATION_ARN"
          value = "${var.aws_organization_arn}"
        },
        {
          name  = "AWS_ORGANIZATION_MASTER_ACCOUNT_ID"
          value = "${var.aws_organization_master_account_id}"
        },
        {
          name  = "AWS_REGION"
          value = "${var.aws_region}"
        },
        {
          name  = "BRANCH_IO_URL"
          value = "${var.branch_io_url}"
        },
        {
          name  = "BRANCH_KEY"
          value = "${var.branch_key}"
        },
        {
          name  = "DATABASE_HOST"
          value = "${module.app_db.db_instance_address}"
        },
        {
          name  = "DATABASE_NAME"
          value = "${module.app_db.db_instance_name}"
        },
        {
          name  = "DATABASE_PORT"
          value = "${module.app_db.db_instance_port}"
        },
        {
          name  = "DATABASE_SCHEMA"
          value = "${var.database_schema}"
        },
        {
          name  = "DATABASE_USER"
          value = "${module.app_db.db_instance_username}"
        },
        {
          name  = "FIREBASE_CLIENT_EMAIL"
          value = "${var.firebase_client_email}"
        },
        {
          name  = "FIREBASE_DATABASE_URL"
          value = "${var.firebase_database_url}"
        },
        {
          name  = "FIREBASE_PROJECT_ID"
          value = "${var.firebase_project_id}"
        },
        {
          name  = "FIREHOSE_NAME"
          value = "${var.firehose_name}"
        },
        {
          name  = "FIREHOSE_ROLE_ARN_TO_ASSUME"
          value = "${var.firehose_role_arn_to_assume}"
        },
        {
          name  = "PORT"
          value = "${local.container_port}"
        },
        {
          name  = "RABBITMQ_USERNAME"
          value = "stagewoodadmin"
        }
      ]
      mount_points = []
      volumes_from = []
      linux_parameters = {
        capabilities = {
          add = []
          drop = [
            "NET_RAW"
          ]
          initProcessEnabled = true
        }
      }
      secrets = [
        {
          name      = "DATABASE_PASSWORD",
          valueFrom = "${module.app_db.db_instance_master_user_secret_arn}:password::"
        },
        {
          name      = "FIREBASE_PRIVATE_KEY",
          valueFrom = "arn:aws:ssm:${var.aws_region}:${local.aws_account_id}:parameter/environment/${var.environment}/FIREBASE_PRIVATE_KEY"
        },
        {
          name      = "RABBITMQ_PASSWORD"
          valueFrom = "arn:aws:secretsmanager:${var.aws_region}:${var.aws_organization_network_account_id}:secret:myn3-broker/stagewoodadmin_password"
        }
      ]
      dependencies = [
        {
          containerName = "fluent-bit"
          condition     = "START"
        }
      ]
      startTimeout              = 30
      stopTimeout               = 120
      user                      = "0"
      privileged                = false
      readonly_root_filesystem  = false
      interactive               = false
      pseudoTerminal            = false
      enable_cloudwatch_logging = true
      log_configuration = {
        logDriver = "awsfirelens"
        options = {
          Name              = "cloudwatch_logs",
          auto_create_group = "true",
          log_group_name    = "/aws/ecs/app/${local.container_name}",
          log_stream_prefix = "app",
          region            = "${var.aws_region}"
        },
      }

      health_check = {
        command = [
          "CMD-SHELL",
          "curl -f http://localhost:${local.container_port}/health || exit 1"
        ]
        interval = 30
        timeout  = 5
        retries  = 3
      }
      system_controls = [
        {
          namespace = "net.ipv4.tcp_keepalive_time"
          value     = 6000
        }
      ]
    }
    # End of application container

    ############################################################################
    # FLUENT BIT CONTAINER FOR LOGS AND ACTIVITY MONITORING

    fluent-bit = {
      name               = "fluent-bit"
      image              = "${var.aws_organization_network_account_id}.dkr.ecr.${var.aws_region}.amazonaws.com/fluentbit:latest"
      cpu                = 512
      memory             = 1024
      memory_reservation = 1024
      port_mappings = [
        {
          containerPort = 2020
          hostPort      = 2020
          protocol      = "tcp"
          appProtocol   = "http"
        },
        {
          containerPort = 24224
          hostPort      = 24224
          protocol      = "tcp"
          appProtocol   = "http"
        }
      ]
      essential = true
      environment = [
        {
          name  = "APP_NAME"
          value = "${local.container_name}"
        },
        {
          name  = "AWS_REGION"
          value = "${var.aws_region}"
        },
        {
          name  = "FIREHOSE_NAME"
          value = "${var.firehose_name}"
        },
        {
          name  = "FIREHOSE_ROLE_ARN_TO_ASSUME"
          value = "${var.firehose_role_arn_to_assume}"
        }
      ]
      mount_points = []
      volumes_from = []
      linux_parameters = {
        capabilities = {
          add                = []
          drop               = []
          initProcessEnabled = true
        }
      }
      startTimeout             = 30
      stopTimeout              = 120
      user                     = "0"
      privileged               = false
      readonly_root_filesystem = true
      interactive              = false
      pseudoTerminal           = false
      log_configuration = {
        logDriver = "awslogs"
        options = {
          awslogs-create-group  = "true"
          awslogs-group         = "/aws/ecs/firelens/${local.container_name}"
          awslogs-region        = "${var.aws_region}"
          awslogs-stream-prefix = "firelens"
        }
      }
      firelens_configuration = {
        type = "fluentbit",
        options = {
          config-file-type        = "file",
          config-file-value       = "/fluent-bit/etc/custom-fluent-bit.conf"
          enable-ecs-log-metadata = "true"
        }
      }
      health_check = {
        command = [
          "CMD-SHELL",
          "curl -f http://127.0.0.1:2020/api/v1/health || exit 1"
        ]
        interval    = 30
        timeout     = 5
        retries     = 3
        startPeriod = 50
      }
      system_controls = []
    }
    # End of fluent-bit container
  }
}

}

tags = merge(local.common_tags, {
Name = "${local.naming_prefix}-app_ecs"
}
)
}

The Container definitions also failing:
##################################################################################

ECS APP CONTAINER DEFINITION

##################################################################################

locals {
app_container_definition = {
name = var.service_name
image = "${module.ecr_app.repository_url}:latest"
cpu = var.app_container_cpu
memory = var.app_container_memory
memory_reservation = var.app_container_memory_reservation
port_mappings = [
{
name = var.service_name
containerPort = var.app_container_port
hostPort = var.app_container_port
protocol = "tcp"
appProtocol = "http"
},
{
name = "${var.service_name}-service_discovery"
containerPort = var.service_discovery_port
hostPort = var.service_discovery_port
protocol = "tcp"
appProtocol = "http"
},
]
essential = true
command = [
"/bin/sh",
"-c",
"/home/node/entrypoint.sh"
],
environment = [
{
name = "APP_NAME"
value = var.service_name
},
{
name = "AWS_ORGANIZATION_ARN"
value = data.aws_organizations_organization.aws_organization.arn
},
{
name = "AWS_ORGANIZATION_MASTER_ACCOUNT_ID"
value = data.aws_organizations_organization.aws_organization.master_account_id
},
{
name = "AWS_REGION"
value = var.aws_region
},
{
name = "BRANCH_IO_URL"
value = var.branch_io_url
},
/*
{
name = "BRANCH_KEY"
value = var.branch_key
},
/
{
name = "DATABASE_HOST"
value = var.db_host
},
{
name = "DATABASE_NAME"
value = var.db_name
},
{
name = "DATABASE_PORT"
value = var.db_port
},
{
name = "DATABASE_SCHEMA"
value = var.service_name
},
{
name = "DATABASE_USER"
value = var.db_user
},
/
{
name = "FIREBASE_CLIENT_EMAIL"
value = var.firebase_client_email
},
/
{
name = "FIREBASE_DATABASE_URL"
value = var.firebase_database_url
},
{
name = "FIREBASE_PROJECT_ID"
value = var.firebase_project_id
},
/
{
name = "FIREBASE_PRIVATE_KEY"
value = var.firebase_private_key
},
/
{
name = "FIREHOSE_NAME"
value = data.terraform_remote_state.central.outputs.firehose_stream_name
},
{
name = "FIREHOSE_ROLE_ARN_TO_ASSUME"
value = data.terraform_remote_state.central.outputs.firehose_iam_role_arn
},
{
name = "PORT"
value = var.app_container_port
},
{
name = "RABBITMQ_USERNAME"
value = data.terraform_remote_state.central.outputs.rabbitmq_username
}
]
mount_points = []
volumes_from = []
linux_parameters = {
capabilities = {
add = []
drop = [
"NET_RAW"
]
initProcessEnabled = true
}
}
/
secrets = [
{
name = "DATABASE_PASSWORD",
valueFrom = var.db_master_user_secret_arn
},
{
name = "RABBITMQ_PASSWORD"
valueFrom = data.terraform_remote_state.central.outputs.rabbitmq_password_secret_arn
}
]
*/
dependencies = [
{
containerName = "fluent-bit"
condition = "START"
}
]
startTimeout = 30
stopTimeout = 120
user = "0"
privileged = false
readonly_root_filesystem = false
interactive = false
pseudoTerminal = false
enable_cloudwatch_logging = true
log_configuration = {
logDriver = "awsfirelens"
options = {
Name = "cloudwatch_logs",
auto_create_group = "true",
log_group_name = "/aws/ecs/app/${var.service_name}",
log_stream_prefix = "app",
region = "${var.aws_region}"
},
}

health_check = {
  command = [
    "CMD-SHELL",
    "curl -f http://localhost:${var.app_container_port}/health || exit 1"
  ]
  interval = 30
  timeout  = 5
  retries  = 3
}
system_controls = [
  {
    namespace = "net.ipv4.tcp_keepalive_time"
    value     = 6000
  }
]

}

FLUENT BIT CONTAINER FOR LOGS AND ACTIVITY MONITORING

fluent_bit_container_definition = {
name = "fluent-bit"
image = "${data.terraform_remote_state.network.outputs.account_id}.dkr.ecr.${var.aws_region}.amazonaws.com/fluent-bit:latest"
cpu = var.fluent-bit_container_cpu
memory = var.fluent-bit_container_memory
memory_reservation = var.fluent-bit_container_memory_reservation
port_mappings = [
{
containerPort = var.fluent-bit_http_listener_port
hostPort = var.fluent-bit_http_listener_port
protocol = "tcp"
appProtocol = "http"
},
{
containerPort = var.fluent-bit_host_port
hostPort = var.fluent-bit_host_port
protocol = "tcp"
appProtocol = "http"
}
]
essential = true
environment = [
{
name = "APP_NAME"
value = var.service_name
},
{
name = "AWS_REGION"
value = var.aws_region
},
{
name = "FIREHOSE_NAME"
value = data.terraform_remote_state.central.outputs.firehose_stream_name
},
{
name = "FIREHOSE_ROLE_ARN_TO_ASSUME"
value = data.terraform_remote_state.central.outputs.firehose_iam_role_arn
}
]
mount_points = []
volumes_from = []
linux_parameters = {
capabilities = {
add = []
drop = []
initProcessEnabled = true
}
}
startTimeout = 30
stopTimeout = 120
user = "0"
privileged = false
readonly_root_filesystem = true
interactive = false
pseudoTerminal = false
log_configuration = {
logDriver = "awslogs"
options = {
awslogs-create-group = "true"
awslogs-group = "/aws/ecs/firelens/${var.service_name}"
awslogs-region = var.aws_region
awslogs-stream-prefix = "firelens"
}
}
firelens_configuration = {
type = "fluent-bit",
options = {
config-file-type = "file",
config-file-value = "/fluent-bit/etc/custom-fluent-bit.conf"
enable-ecs-log-metadata = "true"
}
}
health_check = {
command = [
"CMD-SHELL",
"curl -f http://127.0.0.1:2020/api/v1/health || exit 1"
]
interval = 30
timeout = 5
retries = 3
startPeriod = 50
}
system_controls = []
}
}

Steps to reproduce the behavior:

yes yes just builded the module and have weeks trying to find solution to the errors.

Expected behavior

my ecs to build without errors

Actual behavior

it wont let me deploy

Terminal Output Screenshot(s)

Additional context

[sheikhasim]

+1, facing the same issue

[forma-wolf]

+1 facing same issue

[sfiguereo]

I found the problem on my solution. I was using data resources to fetch in values for environmental variables. The module did not reconized those values as available when deployment. Once I changed the values to set variables the module was able to proceed without the error. Hope this help some other facing the same issue.

[bryantbiggs]

looks like this isn't a module error so closing for now

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.