Skip to content
This repository was archived by the owner on Mar 23, 2026. It is now read-only.
This repository was archived by the owner on Mar 23, 2026. It is now read-only.

bug: Container Health always Unknown #13133

@ErenPhayte

Description

@ErenPhayte

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I was attempting to set up Signoz https://signoz.io/docs/install/ecs/ through Localstack before deploying to AWS.
I wrote this in terraform script. During my initial runs, I was having a few issues - the primary issue was a coordination issue. What I was observing:

Despite clear dependsOn directives, the containers still started up in sequence and didn't obey the dependsOn:

clickhouse container:

dependsOn = [
        {
          containerName = "init-clickhouse"
          condition     = "COMPLETE"
        },
        {
          containerName = "config-fetcher"
          condition     = "COMPLETE"
        },
        {
          containerName = "zookeeper"
          condition     = "HEALTHY"
        }
      ]

schema-migrator-sync

 dependsOn = [
        {
          containerName = "clickhouse"
          condition     = "HEALTHY"
        },
        {
          containerName = "zookeeper"
          condition     = "HEALTHY"
        }
      ]

What was happening:

  • Clickhouse started first instead of waiting till the dependsOn conditions where met.
  • The migration script ran immediate after clickhouse, dispite Zookeeper not starting fully and clickhouse reporting that its connected to zookeeper

Then I noticed, that conditions ALWAYS seemed to be UNKNOWN:

clickhouse container:

healthCheck = {
        command     = ["CMD-SHELL", "wget --spider -q 0.0.0.0:8123/ping || exit 1"]
        interval    = 30
        timeout     = 5
        retries     = 3
        startPeriod = 60
      }

zookeeper container:

healthCheck = {
        command     = ["CMD-SHELL", "curl -s -m 2 http://localhost:8080/commands/ruok | grep error | grep null"]
        interval    = 15
        timeout     = 5
        retries     = 3
        startPeriod = 30
      }

Here I was monitoring the container status:

Every 2,0s: awslocal ecs describe-tasks --cluster signoz-cluster --tasks arn:aws:ecs:us-west-2:000000000000:task/signoz-cluster/c5507729-9612-43df-9621-bbbd8237bbb4 --region=us-we… 
                                                                                                                                                                                                        in 0,512s (0)
----------------------------------------------------
|                   DescribeTasks                  |
+---------+--------------+-------------------------+
| health  | lastStatus   |          name           |
+---------+--------------+-------------------------+
|  UNKNOWN|  STOPPED     |  init-clickhouse        |
|  UNKNOWN|  STOPPED     |  config-fetcher         |
|  UNKNOWN|  RUNNING     |  zookeeper              |
|  UNKNOWN|  RUNNING     |  clickhouse             |
|  UNKNOWN|  STOPPED     |  schema-migrator-sync   |
+---------+--------------+-------------------------+

This is when I noticed that the health status is never updated to anything but unknown, not even status = healthy

Expected Behavior

Expected behaviour is that dependsOn respects the conditions, and the healthStatus respects the healthChecks

How are you starting LocalStack?

Custom (please describe below)

Steps To Reproduce

Create a few containers that connect to each other. Make sure they all have healthchecks and that they depend on condition = "HEALTHY" .. make sure some containers can go from healthy state to unhealthy state.

  • Observe the health status of all containers when they start up, and if they obey the healthChecks
  • Observe any containers that do not spin up because a dependent container didnt start up condition = "UNHEALTHY"
  • Observe start-up sequences of containers based on dependsOn conditions

Not sure if this is important, but I was using the following:

requires_compatibilities = ["EC2"]
network_mode = "awsvpc"

Environment

- OS: Mac
- LocalStack:
  LocalStack version:  4.7.1.dev141
  LocalStack build date:2025-09-10
  LocalStack build git hash: 92a0f67f7

Anything else?

I use the Localstack extension in Docker to start up my environment, below is my configuration (I removed the token from image but I am connecting to Localstack Pro):

Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    aws:ecsAmazon Elastic Container Servicestatus: backlogTriaged but not yet being worked ontype: bugBug report

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions