the `--exit-code-from` is 0 from a service that `depends_on` when another container exited

[Lenormju]

Description

I want to use Compose to test my services. I use --exit-code-from test to get the exit code of my test service, which is very convenient.
As mentioned in the docs, --exit-code-from implies --abort-on-container-exit, which is very useful in my case : if any service crashes, the test can be considered a failure.
But if my test service has a depends_on on any other service, then I don't get the exit code anymore, just 0.

Steps to reproduce the issue:

1. file: docker-compose.yml

   ---
   name: mre__exit_code_from__when_other_container_aborts
   services:
     safe:
       image: 'ubuntu:20.04'
       command: ['/usr/bin/bash', '-c', 'sleep 30']
     failure:
       image: 'ubuntu:20.04'
       command: ['/usr/bin/bash', '-c', 'sleep 2 ; echo "exiting" ; exit 42']
     test:
       image: 'ubuntu:20.04'
       command: ['/usr/bin/bash', '-c', 'sleep 30']
       depends_on: [safe]
   ...

2. run (linux) docker compose down ; docker compose up --abort-on-container-exit --exit-code-from test ; echo "EXIT_CODE=$?" ; docker ps -a --filter 'name=mre__exit_code_from__when_other_container_aborts'

Describe the results you received:

[+] Running 4/3
 ⠿ Network mre__exit_code_from__when_other_container_aborts_default      Created                                                                                                                                                    0.1s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-failure-1  Created                                                                                                                                                    0.1s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-safe-1     Created                                                                                                                                                    0.1s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-test-1     Created                                                                                                                                                    0.1s
Attaching to mre__exit_code_from__when_other_container_aborts-failure-1, mre__exit_code_from__when_other_container_aborts-safe-1, mre__exit_code_from__when_other_container_aborts-test-1
mre__exit_code_from__when_other_container_aborts-failure-1  | exiting
mre__exit_code_from__when_other_container_aborts-failure-1 exited with code 42
Aborting on container exit...
[+] Running 3/3
 ⠿ Container mre__exit_code_from__when_other_container_aborts-test-1     Stopped                                                                                                                                                   10.4s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-failure-1  Stopped                                                                                                                                                    0.0s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-safe-1     Stopped                                                                                                                                                   10.4s
ERRO[0024] 137                                          
ERRO[0024] 0                                            
EXIT_CODE=0
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS                                PORTS     NAMES
c00c28b861ea   ubuntu:20.04   "/usr/bin/bash -c 's…"   24 seconds ago   Exited (137) 10 seconds ago                     mre__exit_code_from__when_other_container_aborts-test-1
d7a924757da7   ubuntu:20.04   "/usr/bin/bash -c 's…"   24 seconds ago   Exited (137) Less than a second ago             mre__exit_code_from__when_other_container_aborts-safe-1
56713d9643b8   ubuntu:20.04   "/usr/bin/bash -c 's…"   24 seconds ago   Exited (42) 21 seconds ago                      mre__exit_code_from__when_other_container_aborts-failure-1

The exit code of the program is 0.

Describe the results you expected:
The exit code of the program is 137.

Additional information you deem important (e.g. issue happens only occasionally):
Reproducible repeatedly with the command I supplied.
Commenting out the depends_on line gets the expected result.
Among the 6 depends_on permutations, only this one exhibits this behavior, the other 5 return 137.
If test finishes early (for example with command: ['/usr/bin/bash', '-c', 'exit 18']) then it does not have to be forced to stop, and the exit code is the expected one.

Output of docker compose version:

docker version
Client: Docker Engine - Community
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:48:02 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:45:53 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker compose version:

Docker Compose version v2.3.3

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
  compose: Docker Compose (Docker Inc., v2.3.3)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 13
  Running: 3
  Paused: 0
  Stopped: 10
 Images: 124
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.11.0-38-generic
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.26GiB
 Name: [REDACTED]
 ID: XJXT:QSMF:UNWH:4VTI:HD5T:MBMN:EZJL:GYQ6:VOPO:2U6Q:3TVF:BQPH
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:
Reproduced on Windows (Docker Desktop).

[Lenormju]

Possibly a duplicate from #9594 (but it did not provide a Minimal Reproducible Example, so I can't tell).

[laurazard]

Hi @Lenormju, thanks for the report!

Trying to figure out what is happening, it seems that adding the depends_on: [safe] to your test service is making it so that the test container has enough time for sleep 30 to execute and exit with EXIT_CODE=0.

Removing the depends_on makes it exit with EXIT_CODE=137 because Compose isn't waiting for safe to stop before killing test. This is verifiable by changing your file such as:

services:
  safe:
    image: 'ubuntu:20.04'
    command: ['/usr/bin/bash', '-c', 'sleep 10']        # changed 30 -> 10
  failure:
    image: 'ubuntu:20.04'
    command: ['/usr/bin/bash', '-c', 'sleep 2 ; echo "exiting" ; exit 42']
  test:
    image: 'ubuntu:20.04'
    command: ['/usr/bin/bash', '-c', 'sleep 30']
    depends_on: [safe]

This way, even with depends_on, we get EXIT_CODE=137 because since safe has already exited way before test's sleep 30 will finish, Compose is forced to kill test resulting in a 137 exit code.

The exit code from the test container is being correctly reported in both cases.

[laurazard]

I'm closing the issue, but feel free to reopen if there is any additional information I'm missing or if I misunderstood the issue. Additionally, if you only care that any container exits, you can just use --abort-on-container-exit. With your example file, this would return an exit code of 42 (from your failure service).

[Lenormju]

Indeed, but safe is not supposed to exit early. I'm not looking for a quick fix like you suggested, I already have one : checking docker inspect mre__exit_code_from__when_other_container_aborts-test-1 --format '{{.State.ExitCode}}'.
I genuinely think there is a problem with the exit code that Compose is returning : when there is no depends_on, I get a 137, but if test has a depends_on another service, then it is 0, although it got aborted. In every other combination, I get the container exit_code, but not this one.

@laurazard I seem to be unable to reopen it.

[laurazard]

@Lenormju I don't understand what you think is incorrect:

When there is no depends_on, failure fails early and due to --abort-containers-on-exit, the other containers are killed. Since test is stuck running sleep 30, Compose is forced to SIGKILL it, resulting on the 137 exit code.

When there is a depends_on, the order in which Compose kills containers is altered: Compose kills safe first, and then tries to stop test. However, with your example, this affords test enough time to exit with exit code 0 before Compose is forced to kill it, which is the exit code that should be reported since we are running with --exit-code-from test.

If there is a depends_on but the service that is depended on has already exited, not affording any extra time, then Compose is forced to kill test, resulting in the 137 exit code again. What depends_on is changing here in this constrained example is whether test has enough time to exit properly exit code 0 or Compose is forced to kill it exit code 137, which changes it's exit code.

[Lenormju]

I do understand that. I mentioned exactly that in the "Additional information" :

If test finishes early [...] then it does not have to be forced to stop, and the exit code is the expected one.

What I want to report is that in the case that test was still running and gets stopped, although the test container's exit code was not zero, the Compose exit code is 0.
Here is a modified version :

• sleep durations : safe is never exiting, it has to get stopped, and test takes a very long time, much more than the timeout, so it has to be stopped too.
• test would print "tests are OK" if it ends

---
name: mre__exit_code_from__when_other_container_aborts
services:
  safe:
    image: 'ubuntu:20.04'
    command: ['/usr/bin/bash', '-c', 'sleep infinity']  # never exiting
  failure:
    image: 'ubuntu:20.04'
    command: ['/usr/bin/bash', '-c', 'sleep 2 ; echo "exiting" ; exit 42']
  test:
    image: 'ubuntu:20.04'
    command: ['/usr/bin/bash', '-c', 'sleep 99999 ; echo "tests are OK"']  # very long job
    depends_on: [safe]
...

$ docker compose down ; docker compose up --abort-on-container-exit --exit-code-from test ; echo "EXIT_CODE=$?" ; docker ps -a --filter 'name=mre__exit_code_from__when_other_container_aborts'

[+] Running 4/4
 ⠿ Network mre__exit_code_from__when_other_container_aborts_default      Created           0.1s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-safe-1     Created           0.1s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-failure-1  Created           0.1s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-test-1     Created           0.1s
Attaching to mre__exit_code_from__when_other_container_aborts-failure-1, mre__exit_code_from__when_other_container_aborts-safe-1, mre__exit_code_from__when_other_container_aborts-test-1

mre__exit_code_from__when_other_container_aborts-failure-1  | exiting
mre__exit_code_from__when_other_container_aborts-failure-1 exited with code 42

Aborting on container exit...
[+] Running 3/3
 ⠿ Container mre__exit_code_from__when_other_container_aborts-test-1     Stopped          10.4s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-failure-1  Stopped           0.0s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-safe-1     Stopped          10.5s

As soon as failure exits, Compose starts to wait for test to stop for up to ~10 seconds (the timeout default duration), then pass over failure (already stopped due to exiting, which caused the abort), and finally get to wait up for safe too.
What bothers me is that Docker prints these two lines next :

ERRO[0024] 137                                          
ERRO[0024] 0     

It seems to override the 137 exit code (test having been stopped), and replace it with 0. Compose exits and the process exit code is 0.

EXIT_CODE=0

Then if i look at the containers, for test its exit code is 137.

CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS                                PORTS     NAMES
933750dcce4a   ubuntu:20.04   "/usr/bin/bash -c 's…"   24 seconds ago   Exited (137) 10 seconds ago                     mre__exit_code_from__when_other_container_aborts-test-1
4bd07d21dd90   ubuntu:20.04   "/usr/bin/bash -c 's…"   24 seconds ago   Exited (42) 21 seconds ago                      mre__exit_code_from__when_other_container_aborts-failure-1
ba3aa232c37e   ubuntu:20.04   "/usr/bin/bash -c 's…"   24 seconds ago   Exited (137) Less than a second ago             mre__exit_code_from__when_other_container_aborts-safe-1

So --exit-code-from test did not return me the test exit code. That is my problem.

The same thing but if there is no depends_on :

---
name: mre__exit_code_from__when_other_container_aborts
services:
  safe:
    image: 'ubuntu:20.04'
    command: ['/usr/bin/bash', '-c', 'sleep infinity']  # never exiting
  failure:
    image: 'ubuntu:20.04'
    command: ['/usr/bin/bash', '-c', 'sleep 2 ; echo "exiting" ; exit 42']
  test:
    image: 'ubuntu:20.04'
    command: ['/usr/bin/bash', '-c', 'sleep 99999 ; echo "tests are OK"']  # very long job
    #depends_on: [safe]
...

[...]

mre__exit_code_from__when_other_container_aborts-failure-1  | exiting
mre__exit_code_from__when_other_container_aborts-failure-1 exited with code 42

Aborting on container exit...
[+] Running 3/3
 ⠿ Container mre__exit_code_from__when_other_container_aborts-failure-1  Stopped           0.0s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-safe-1     Stopped          10.5s
 ⠿ Container mre__exit_code_from__when_other_container_aborts-test-1     Stopped          10.5s

ERRO[0013] 137           
                               
EXIT_CODE=137

CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS                                PORTS     NAMES
e291b1af8ebe   ubuntu:20.04   "/usr/bin/bash -c 's…"   14 seconds ago   Exited (137) Less than a second ago             mre__exit_code_from__when_other_container_aborts-safe-1
3928efd96fd9   ubuntu:20.04   "/usr/bin/bash -c 's…"   14 seconds ago   Exited (137) Less than a second ago             mre__exit_code_from__when_other_container_aborts-test-1
6c1aa2305754   ubuntu:20.04   "/usr/bin/bash -c 's…"   14 seconds ago   Exited (42) 10 seconds ago                      mre__exit_code_from__when_other_container_aborts-failure-1

The containers test and safe are waited in parallel (because there is no depends_on between them) and the exit code is 137 (as expected).

In both cases (with or without depends_on), the test container had to be stopped. But they don't return the same exit code.

If you prefer, I can describe my real setup. I use Compose for integration testing : it starts all the application services, my test service depends_on the frontend. The test service does its job, then it exits with 0 (success) or 1 failure), and my CI uses that to know if the pipeline succeeded or not. Thus, it is important for me that I get the test exit code. If one of my application services crashes, the Compose stops (due to --exit-on-container-abort) which is fine to me, because it stops the test container (and return an exit code != 0). But I got undetected errors because sometimes one application service crashes, so Compose stops them all, but I get 0 as exit code instead of 137 (stopped).

If one service crashes, but just a few seconds after that test ends succesfully, so its exit code is legitimately 0, I accept that.
But that is not the case in this issue. One service crashes, the test container get forcefully stopped, and Compose do not return the appropriate exit code.

[laurazard]

@Lenormju thanks for the additional information. I'll look into it some more and report findings :)

[Lenormju]

@laurazard the explanation you provided in the pull request matches the symptom ERRO[0024] 137 \n ERRO[0024] 0 and so I'm confident you have fixed it. Do you know when to expect the next release of Docker Compose ? Thanks very much anyway 😸

[Lenormju]

It is fixed in my current version (Docker Compose version v2.16.0, but probably have been well before that).
Thanks again !