Restarting docker causes incorrect exit codes to be reported for the containers #31262

yujuhong · 2017-02-22T20:58:53Z

Description

Docker reports incorrect exit codes (0) for containers after being restarted.

Steps to reproduce the issue:

Start N containers
Restart docker with sudo service docker restart or sudo systemctl restart docker
List all containers with docker ps -a

Describe the results you received:

$ sudo docker ps -a
CONTAINER ID        IMAGE                                      COMMAND                   CREATED             STATUS                      PORTS               NAMES
d30de3f8c5ad        busybox                                    "sleep 1000"              4 minutes ago       Exited (0) 3 minutes ago                        sharp_jennings
2e3449bf325a        busybox                                    "sleep 1000"              4 minutes ago       Exited (0) 3 minutes ago                        romantic_meitner
6808f4f189c6        busybox                                    "sleep 1000"              4 minutes ago       Exited (0) 3 minutes ago                        goofy_kowalevski
ac30162ecce3        busybox                                    "sleep 1000"              4 minutes ago       Exited (0) 3 minutes ago                        grave_allen
4d3264ad0aaa        busybox                                    "sleep 1000"              4 minutes ago       Exited (0) 3 minutes ago                        stupefied_austin

Describe the results you expected:

The exit code should be non-zero.

Output of docker 1.10.3:

$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                    PORTS               NAMES
8d98beb00f26        busybox             "/bin/sh -c 'sleep 10"   6 days ago          Exited (137) 6 days ago                       elegant_carson
a6cd23bd5fda        busybox             "/bin/sh -c 'sleep 10"   6 days ago          Exited (137) 6 days ago                       dreamy_cray
12ae2eb58c96        busybox             "/bin/sh -c 'sleep 10"   6 days ago          Exited (137) 6 days ago                       gloomy_feynman
8816d3b4509d        busybox             "/bin/sh -c 'sleep 10"   6 days ago          Exited (137) 6 days ago                       pedantic_wescoff
d410a834df2d        busybox             "/bin/sh -c 'sleep 10"   6 days ago          Exited (137) 6 days ago                       mad_darwin

Additional information you deem important (e.g. issue happens only occasionally):

I tested this on 3 versions: 1.10.3, 1.11.2, and 1.12.6, and only 1.10.3 reported the correct exit codes.
#26143 reported the same problem for 1.12.1, but was closed with the fix #26158. #26158 (comment) also reported the same problem in 1.12.3.

Output of docker version:

Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:26:30 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:26:30 2017
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 35
 Running: 0
 Paused: 0
 Stopped: 35
Images: 2
Server Version: 1.12.6
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 74
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host null bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 4.4.0-59-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.613 GiB
Name: docker1-12
ID: L4UT:MY3O:OYBF:276K:HOWZ:BEY5:Y2EY:ZEEC:QOJS:5ARZ:G3NM:D2DG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

The text was updated successfully, but these errors were encountered:

joequery · 2017-02-23T14:46:20Z

Using Docker version 1.13.1, I could not replicate the issue directly.

[DevVM]~/Sandboxes/Docker/tmptest$ docker run -d -i ubuntu /bin/bash
7c66d43c1e5da26b196a80e8cf29e419271b2eff9fa9e854746f3746265fe828
[DevVM]~/Sandboxes/Docker/tmptest$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
7c66d43c1e5d        ubuntu              "/bin/bash"         5 seconds ago       Up 5 seconds                            modest_panini
[DevVM]~/Sandboxes/Docker/tmptest$ sudo service docker restart
docker stop/waiting
docker start/running, process 31967
[DevVM]~/Sandboxes/Docker/tmptest$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                       PORTS               NAMES
7c66d43c1e5d        ubuntu              "/bin/bash"         32 seconds ago      Exited (137) 5 seconds ago                       modest_panini

However, if I run the container and attach to it (docker run -i -t ubuntu /bin/bash) and then restart the docker service in another tab, I do see the Exit 0.

[DevVM]~/Sandboxes/Docker/tmptest$ docker run -i -t ubuntu /bin/bash
root@19bb80d49bd0:/#

Separate tab:

DevVM]~/Sandboxes/Docker/compose$ sudo service docker restart
docker stop/waiting
docker start/running, process 648
[DevVM]~/Sandboxes/Docker/compose$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
19bb80d49bd0        ubuntu              "/bin/bash"         27 seconds ago      Exited (0) 3 seconds ago                       gallant_jennings

I'm not sure if that is expected behavior.

cpuguy83 · 2017-02-23T14:49:25Z

I'm not seeing this behavior on 1.13.1, however is reproducible on 1.12.6.

Can you check on 1.13.1?

yujuhong · 2017-02-23T18:14:35Z

@cpuguy83 I can believe that this is no longer an issue in 1.13, but is it possible to fix this in 1.12? We need more time to qualify docker 1.13 for kubernetes, and users are hitting this issue.

cpuguy83 · 2017-02-23T18:16:21Z

@yujuhong Unfortunately no, we are not doing any new releases for 1.12.

yujuhong · 2017-02-23T18:52:15Z

It was worth a try... Thanks.

Random-Liu · 2017-04-05T01:27:09Z

I could still reproduce this easily in 17.03.1-ce.

Docker version:

$ sudo docker version
Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:10:36 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:10:36 2017
 OS/Arch:      linux/amd64
 Experimental: false

OS:

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.5 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

Start a nginx container:

$ sudo docker run -d nginx
108ee8a27081e24fdebeac8408fe3255e814024e610b8916ad091cb7bae1630c
$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
108ee8a27081        nginx               "nginx -g 'daemon ..."   13 seconds ago      Up 12 seconds       80/tcp, 443/tcp     elated_snyder

Restart docker daemon:

$ sudo service docker restart
docker stop/waiting
docker start/running, process 21525
$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS                        PORTS               NAMES
108ee8a27081        nginx               "nginx -g 'daemon ..."   About a minute ago   Exited (0) 17 seconds ago                         elated_snyder

cpuguy83 · 2017-04-05T01:34:10Z

@Random-Liu this looks correct. Containers are shutdown when you stop docker by default.

cpuguy83 · 2017-04-05T01:34:48Z

Closing as this appears to have been resolved in 1.13.

Random-Liu · 2017-04-05T04:01:33Z

@cpuguy83 You are correct. I tried docker run -d ubuntu sleep 10000, and it does generate exit code 137.

Maybe nginx handles SIGTERM gracefully, and generates exit code 0.

GordonTheTurtle added the version/1.12 label Feb 22, 2017

yujuhong mentioned this issue Feb 22, 2017

Docker reports incorrect exit code for containers if the docker service was restarted kubernetes/kubernetes#41516

Closed

cpuguy83 closed this as completed Apr 5, 2017

xiang90 mentioned this issue May 26, 2017

on all nodes restart cluster stays in completed state coreos/etcd-operator#1130

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restarting docker causes incorrect exit codes to be reported for the containers #31262

Restarting docker causes incorrect exit codes to be reported for the containers #31262

yujuhong commented Feb 22, 2017

joequery commented Feb 23, 2017

cpuguy83 commented Feb 23, 2017

yujuhong commented Feb 23, 2017

cpuguy83 commented Feb 23, 2017

yujuhong commented Feb 23, 2017

Random-Liu commented Apr 5, 2017

cpuguy83 commented Apr 5, 2017

cpuguy83 commented Apr 5, 2017

Random-Liu commented Apr 5, 2017

Restarting docker causes incorrect exit codes to be reported for the containers #31262

Restarting docker causes incorrect exit codes to be reported for the containers #31262

Comments

yujuhong commented Feb 22, 2017

joequery commented Feb 23, 2017

cpuguy83 commented Feb 23, 2017

yujuhong commented Feb 23, 2017

cpuguy83 commented Feb 23, 2017

yujuhong commented Feb 23, 2017

Random-Liu commented Apr 5, 2017

cpuguy83 commented Apr 5, 2017

cpuguy83 commented Apr 5, 2017

Random-Liu commented Apr 5, 2017