Wait.forHealthCheck timout, but docker ps says "healthy"?! 🤔

[mkarg]

I have set up a simple GenericContainer, and try to wait for it to finish booting.

Using Wait.forLogMessage works pretty well, but Wait.forHealthCheck always runs into timeout. One thinks that the container would not become healthy within one minute, so I set up the exact same image with exact same configuration manually using docker run, and guess what, docker ps says healthy! 😲 I even used .withLogConsumer to "see" what is going on, and the container really says that it has booted successfully and is healthy! 🤯

This is perfectly reproducible on my laptop, and I am really running out of ideas why Testcontainers thinks the container is not healthy?! 🤔

[g4lb]

Check the HEALTHCHECK interval of the image:

docker inspect --format '{{json .Config.Healthcheck}}' <image>

Docker runs the first check only after one interval. Wait.forHealthcheck() stops after 60 seconds by default. If the interval is longer, the container becomes healthy too late. docker ps shows "healthy" because you look at it later.

Set a longer timeout:

.waitingFor(Wait.forHealthcheck().withStartupTimeout(Duration.ofMinutes(3)))

I tested this with Testcontainers 2.0.5 and HEALTHCHECK --interval=70s. With the default timeout, the start failed after 60 seconds. With a 2-minute timeout, the container was healthy after 78 seconds.

[mkarg]

I am using .waitingFor(Wait.forHealthcheck().withStartupTimeout(Duration.ofMinutes(3))); with HEALTHCHECK --start-period=5s CMD ..., and manual call of docker reports healthy after five seconds, while Testcontainers still reports timeout after 3 minutes.

[g4lb]

I tested --start-period=5s with Testcontainers 2.0.5. The container was healthy after 7 seconds.

While Testcontainers waits, please send the output of:

docker inspect --format '{{json .State.Health}}' <container-id>
docker version

[mkarg]

Thank you for your kind help so far. 👍

In fact the problem has gone meanwhile. It simply was a port mismatch within the container image itself, which happend only under very specific cases - which I happend to have in my specific test. Nothing of the chain of effects was related to or actually caused by Testcontainers.

Having said that, closing the discussion hereby.

Again, thanks for everything! ❤️

[niranjankagri]

I tried to reproduce this with Testcontainers 2.0.5 on Docker 29.8.2 (Windows, Docker Desktop), and Wait.forHealthcheck() itself works with a start period:

HEALTHCHECK options in the image	healthy after
--start-period=5s (interval left at the default 30s)	16.4 s
--start-period=5s --interval=2s	7.2 s
--start-period=5s --start-interval=1s	3.6 s

So my guess is that the container Testcontainers starts is not the same as the one from your docker run. One case fits your symptom exactly: when the container has no health check at all, Wait.forHealthcheck() does not tell you so. It just waits and fails with "Timed out waiting for container to become healthy". That happens when the check only exists as --health-cmd flags on your docker run command line, or in a compose file, but not as a HEALTHCHECK in the image.

The quickest way to tell is to look at the Testcontainers container while it is still waiting:

docker ps                      # find the container Testcontainers started
docker inspect --format '{{json .State.Health}}' <container-id>

• null: this container has no health check. Add it to the image, or set it from code:

  .withCreateContainerCmdModifier(cmd -> cmd.withHealthcheck(new HealthCheck()
      .withTest(List.of("CMD-SHELL", "your-check"))
      .withInterval(Duration.ofSeconds(2).toNanos())       // docker-java expects nanoseconds
      .withStartPeriod(Duration.ofSeconds(5).toNanos())))

  (Passing plain seconds, e.g. withInterval(2L), fails right away with Status 400: Interval in Healthcheck cannot be less than 1ms.)
• "Status":"starting" or "unhealthy": the Log array shows the exit code and output of each check. That tells you what the check sees inside this container that differs from your manual run (environment variables, command, user, network).

If it turns out to be something else, the output of that docker inspect would help narrow it down.