Healthcheck when running in docker

[use-strict]

I'm running localstack with docker-compose as a dependency for a service that I'm developing. The problem is that my service calls localstack before it's fully initialized. The only solution I could find so far is a hard sleep <seconds> at start-up, but that only works on my specific system and produces unexpected results for other developers. Can localstack expose a healthcheck, so I can have docker-compose start my service after localstack is "healthy"?

A trimmed down version of my docker-compose.yml looks something like this:

myservice:
  command: "sh -c 'sleep 10 && npm run start'" #grrrrr
  depends_on:
    - aws
    # aws:
    #   condition: service_healthy
aws:
  image: localstack/localstack
  environment:
    SERVICES: s3:81,sqs:82,ses:83
    HOSTNAME_EXTERNAL: aws

[ysamlan]

FYI healthcheck dependency conditions won't work on docker-compose v3 anyways, so you may be better off hard coding a check into your other service's init scripts.

I just stick this in my dependent service's init script:

until awslocal cloudformation list-stacks
do
  echo "waiting for localstack cloudformation…"
  sleep 3
done

but you could probably also run something similar for any of the services it provides directly as the healthcheck for the localstack image itself (either dockerfile yourself a new layer, or you could probably stick it directly onto the localstack config in your docker-compose.yml, although you will have to do pip install awscli-local first).

[use-strict]

Thank you.

you may be better off hard coding a check into your other service's init scripts.

That would mean installing awslocal in every image that depends on localstack. This is not scalable.

I'm using docker-compose v2 format. I managed to create a custom Dockerfile based on localstack w/ awslocal installed. I ran your command as a HEALTHCHECK and it seems to be working. One problem though: docker-compose only starts producing output after the service is healthy, so I don't see that "echo" or anything else during start-up. Initially, for example, I forgot to add "cloudformation" to the SERVICES env and it was just hanging with no console output.

[giorgiosironi]

I'm not sure what awslocal is, but I implemented healthcheck in the docker-compose.yml like this:

healthcheck:
    test: "bash -c 'AWS_ACCESS_KEY_ID=fake AWS_SECRET_ACCESS_KEY=fake aws --endpoint-url=http://localhost:4572 s3 ls'"

(for s3 usage; customize to your favorite services)

[terlar]

Since the container for localstack already contain the awslocal tool you can do something like this:

    healthcheck:
      test:
        - CMD
        - bash
        - -c
        - awslocal dynamodb list-tables
          && awslocal es list-domain-names
          && awslocal s3 ls
          && awslocal sqs list-queues
      interval: 5s
      timeout: 10s
      start_period: 10s

[whummer]

Looks like some feasible workarounds have been posted for this issue. Thanks all, for providing your solutions. 👍

[jkentjnr]

This is a useful command if you are waiting for a S3 bucket to be created as part of the LocalStack init:
aws --endpoint-url=http://localhost:4566 s3api wait bucket-exists --bucket localstack-dropbox-dev

[jaecktec]

since this is the first answer on google:

test for the new health endpoint:

curl -s http://localhost:4566/_localstack/health | jq 'all(.services | to_entries | .[] .value; . as $a | ["running", "available"] | index($a))'

[rquadling]

@jaecktec , I'm getting this message ...

WARN[0000] The "a" variable is not set. Defaulting to a blank string. 
WARN[0000] The "a" variable is not set. Defaulting to a blank string. 

twice every time. The $a is being read as a variable somehow.

But that's not the full issue.

If my healthcheck test a super dumb

    test: aws --version && jq --version

and I run

$ docker inspect --format "{{json .State.Health }}" localstack_main | jq .

after loading the stack.

I'm getting

{
  "Status": "starting",
  "FailingStreak": 4,
  "Log": [
    {
      "Start": "2024-02-21T11:07:12.941740438Z",
      "End": "2024-02-21T11:07:13.527845374Z",
      "ExitCode": 127,
      "Output": "aws-cli/1.30.5 Python/3.11.6 Linux/6.6.12-linuxkit botocore/1.32.5\n/bin/sh: 1: jq: not found\n"
    },
    {
      "Start": "2024-02-21T11:07:18.529702367Z",
      "End": "2024-02-21T11:07:19.119717121Z",
      "ExitCode": 127,
      "Output": "aws-cli/1.30.5 Python/3.11.6 Linux/6.6.12-linuxkit botocore/1.32.5\n/bin/sh: 1: jq: not found\n"
    },
    {
      "Start": "2024-02-21T11:07:24.121755745Z",
      "End": "2024-02-21T11:07:24.599580773Z",
      "ExitCode": 127,
      "Output": "aws-cli/1.30.5 Python/3.11.6 Linux/6.6.12-linuxkit botocore/1.32.5\n/bin/sh: 1: jq: not found\n"
    },
    {
      "Start": "2024-02-21T11:07:29.602651314Z",
      "End": "2024-02-21T11:07:30.13433051Z",
      "ExitCode": 127,
      "Output": "aws-cli/1.30.5 Python/3.11.6 Linux/6.6.12-linuxkit botocore/1.32.5\n/bin/sh: 1: jq: not found\n"
    },
    {
      "Start": "2024-02-21T11:07:35.1348704Z",
      "End": "2024-02-21T11:07:35.652921213Z",
      "ExitCode": 127,
      "Output": "aws-cli/1.30.5 Python/3.11.6 Linux/6.6.12-linuxkit botocore/1.32.5\n/bin/sh: 1: jq: not found\n"
    }
  ]
}

AWS CLI is available, but jq is not. So running jq in the healthcheck isn't going to work. jq is installed on my host.

Oh. As a test, I added echo pwd and the output path is /opt/code which is not on my host anywhere.

So I am probably missing something super obvious. I'm on a Mac, and that's probably different enough to mess things up.

[rquadling]

Me too! jq IS installed. I set my test to be uname -a.

{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2024-02-21T11:27:10.68585453Z",
      "End": "2024-02-21T11:27:11.354797815Z",
      "ExitCode": 0,
      "Output": "Linux e5b12ef2e021 6.6.12-linuxkit #1 SMP PREEMPT_DYNAMIC Tue Jan 30 09:48:40 UTC 2024 x86_64 GNU/Linux\n"
    },
    {
      "Start": "2024-02-21T11:27:16.357469254Z",
      "End": "2024-02-21T11:27:17.11303262Z",
      "ExitCode": 0,
      "Output": "Linux e5b12ef2e021 6.6.12-linuxkit #1 SMP PREEMPT_DYNAMIC Tue Jan 30 09:48:40 UTC 2024 x86_64 GNU/Linux\n"
    },
    {
      "Start": "2024-02-21T11:27:22.115897924Z",
      "End": "2024-02-21T11:27:22.841087367Z",
      "ExitCode": 0,
      "Output": "Linux e5b12ef2e021 6.6.12-linuxkit #1 SMP PREEMPT_DYNAMIC Tue Jan 30 09:48:40 UTC 2024 x86_64 GNU/Linux\n"
    },
    {
      "Start": "2024-02-21T11:27:27.843111427Z",
      "End": "2024-02-21T11:27:30.670417094Z",
      "ExitCode": 0,
      "Output": "Linux e5b12ef2e021 6.6.12-linuxkit #1 SMP PREEMPT_DYNAMIC Tue Jan 30 09:48:40 UTC 2024 x86_64 GNU/Linux\n"
    },
    {
      "Start": "2024-02-21T11:27:35.671995864Z",
      "End": "2024-02-21T11:27:36.565221892Z",
      "ExitCode": 0,
      "Output": "Linux e5b12ef2e021 6.6.12-linuxkit #1 SMP PREEMPT_DYNAMIC Tue Jan 30 09:48:40 UTC 2024 x86_64 GNU/Linux\n"
    }
  ]
}

compared to

Darwin Richards-MacBook-Pro.local 21.6.0 Darwin Kernel Version 21.6.0: Thu Nov  9 00:38:19 PST 2023; root:xnu-8020.240.18.705.10~1/RELEASE_X86_64 x86_64

I wondered if pathing was the issue. It is not as the healthcheck is running within the container. I've confirmed that this is also the same error on a Linux machine.

[rquadling]

As an answer to the original question ... what I've found ...

If you have:

        # Using the long syntax to support 'service_healthy' where appropriate : https://github.com/compose-spec/compose-spec/blob/master/spec.md#long-syntax-1
        depends_on:
            localstack:
                condition: service_healthy

then the localstack will need a healthcheck:

        healthcheck:
            # Run the AWS service list-xxx command for each required service for dependants on localstack.
            # One thing to keep in mind is the interval and timeout of 5s each may eventually be too small if your
            # machine is overly busy or you need more services on localstack. Running :
            #   docker inspect --format "{{json .State.Health }}" localstack_main | jq .
            # in a separate window will show you what is going on at the time for the healtchecks on localstack.
            test: >-
                awslocal sqs list-queues && 
                awslocal iam list-users && 
                awslocal lambda list-functions
            interval: 5s
            timeout: 5s
            retries: 10

We have about 5 lambda functions. Without this setup, they always complained/failed to load as the localstack was not ready.

With this setup, (and if you run the docker inspect command), we saw that whilst the localstack was not marked as healthy (yes, it had started, but wasn't "healthy"), the lambda functions were sat there waiting to start when running the docker-compose command.

After a few seconds, they came online and all was good.

NOTE: you cannot use the same condition on containers that do not have a healthcheck. That caught us out!

[lrk]

since this is the first answer on google:

test for the new health endpoint:

curl -s http://localhost:4566/_localstack/health | jq 'all(.services | to_entries | .[] .value; . as $a | ["running", "available"] | index($a))'

I did some tests, and the following command might be more specific if you only want enabled services in the docker container:

.venv/bin/localstack status services --format=json | jq --argjson services $(echo $SERVICES | sed 's/\(.*\)/["\1"]/;s/,/","/g') '. as $source | all($services[]; . as $service | $source[$service] == "running")'

• echo $SERVICES | sed 's/\(.*\)/["\1"]/;s/,/","/g' -> will produce a string array based on the comma separated list of services names stored in the SERVICES environment variable. The string array is then pass to jq as json argument identified as "services"
• . as $source | all($services[]; . as $service | $source[$service] == "running") -> . as $source will store the original jq input under the identifier source for later use in the all condition. all($services[]; . as $service | $source[$service] == "running") call the all function with the generator $services[] to iterate over each $services entries. . as $service | $source[$service] == "running" the $source attribute indexed by $service should be equal to "running".

example:

if the ouput of .venv/bin/localstack status services --format=json is:

{
  "acm": "disabled",
  "apigateway": "disabled",
  "cloudformation": "disabled",
  "cloudwatch": "disabled",
  "config": "disabled",
  "dynamodb": "running",
  "dynamodbstreams": "available",
  "secretsmanager": "running",
  ...
}

and SERVICES=dynamodb,secretsmanager then

echo $SERVICES | sed 's/\(.*\)/["\1"]/;s/,/","/g' will output ["dynamodb","secretsmanager"] and jq will output true

if SERVICES=dynamodb,secretsmanager,acm then jq will output false