[BUG] 'docker compose run' hanging if dependent container fails to start

[fgstewart]

Description

When a docker-compose container depends on another using depends_on and waits for service_healthy... if that dependent container exits with an error on startup then the container waiting will wait indefinitely.

I would expect the docker compose run to exit informing the user that a dependent container failed to start. Similar to what happens if the dependent container gets as far as running the health check and is found to be unhealthy then the compose command does in fact exit with an error (as I'd expect).

If this is expected behaviour, is there any better way to detect the failed container and exit the compose command?

Steps To Reproduce

1. The docker-compose.yml demonstrates the issue / observation. It has a test-runner container which depends on two other containers starting up healthy. The frontend starts up ok, the backend is set to exit 1 on startup.

services:

  test-runner:
    image: alpine:latest
    command: sh -c "echo frontend came up"
    depends_on:
      frontend:
        condition: service_healthy
      backend:
        condition: service_healthy

  # will start up healthy
  frontend:
    image: nginx:latest
    ports:
    - 80:80
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost || exit 1"]
      interval: 5s
      timeout: 5s
      retries: 10
      start_period: 60s

  # will exit with error on startup
  backend:
    image: nginx:latest
    ports:
    - 81:80
    command: sh -c "exit 1" # simulate app crash on startup
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost || exit 1"]
      interval: 5s
      timeout: 5s
      retries: 10
      start_period: 60s

2. Run this command:

$ docker compose run test-runner

3. Observe that frontend starts ok, backend exits immediately and that the docker compose run does not return and blocks indefinitely.

Compose Version

Docker Compose version v2.12.2

docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Docker Environment

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.12.2)
  dev: Docker Dev Environments (Docker Inc., v0.0.3)
  extension: Manages Docker extensions (Docker Inc., v0.2.13)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
 Containers: 19
  Running: 1
  Paused: 0
  Stopped: 18
 Images: 134
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1c90a442489720eec95342e1789ee8a5e1b9536f
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 10
 Total Memory: 7.772GiB
 Name: docker-desktop
 ID: FXDS:KFHH:IB6U:2VCG:CMX2:6TZY:5BGZ:KDJ7:IT2V:5ZJP:32TN:QB4J
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[laurazard]

Hiya @fgstewart, thanks for the report. I believe this was fixed in #10029, and in v2.14.2 I can't replicate your issue. I'll close the issue for now, but please comment here if you still experience it with the latest Compose version