Try to avoid issues when the Docker daemon restarts or stops on RHEL/CentOS 6

[smerrill]

This change will allow the Docker daemon's init script to wait up to 5
minutes before being forcibly terminated by the initscript. Many
non-trivial containers will take more than the default 3 seconds to
stop, which can result in containers whose rootfs is still mounted and
will not restart when the daemon starts up again, or worse, orphan
processes that are still running.

[smerrill]

Component

docker-io-1.1.2-1.el6.x86_64

Hardware Platform

x86_64

Platform

CentOS 6.5

Summary

On restart/stop, docker can lose track of currently mounted dm devices on long-running container shutdowns.

Details

The docker daemon can lose track of thin mounted device mapper volumes if the docker daemon is killed. The RHEL 6 initscript uses killproc -p to stop the Docker daemon (see https://github.com/docker/docker/blob/v1.1.2/contrib/init/sysvinit-redhat/docker#L71) This can occur if a process takes longer than the killproc timeout to exit, or if the docker daemon receives a SIG_KILL. When the daemon is restarted, docker will be unable to restart any containers that were still active after the docker daemon dies.

Reproducibility

Every time

Steps to Reproduce

Given a CentOS 6.5 instance with docker-io-1.1.2 installed:

service docker start
docker run -d --name=dockerbug centos:centos6 /bin/sleep 1000
service docker restart
docker start dockerbug

Actual Results

What happened when you reached the bug?

[root@host ~]# docker ps | grep dockerbug
# (no response, this container is not running)
[root@host ~]# docker start dockerbug
Error response from daemon: Cannot start container dockerbug: Error getting container b4fb0bd2b919b4136ee15cf95bfe5e72d5e33c78bfa178bc21404f182c9fc64d from driver devicemapper: Error mounting '/dev/mapper/docker-8:3-12846136-b4fb0bd2b919b4136ee15cf95bfe5e72d5e33c78bfa178bc21404f182c9fc64d' on '/var/lib/docker/devicemapper/mnt/b4fb0bd2b919b4136ee15cf95bfe5e72d5e33c78bfa178bc21404f182c9fc64d': device or resource busy
2014/09/17 16:02:55 Error: failed to start one or more containers

Expected Results

What do you think was supposed to happen?

[root@host ~]# docker ps | grep dockerbug
# (no response, this container is not running)
[root@host ~]# docker start dockerbug
dockerbug

Workaround

After verifying the contained processes have exited, identify the affected device mapper ID by using docker inspect <containername>, and unmount the device by hand. By default it will be /var/lib/docker/devicemapper/mnt/$CONTAINER_ID. You should be able to start the container again.

[root@host ~]# umount /var/lib/docker/devicemapper/mnt/b4fb0bd2b919b4136ee15cf95bfe5e72d5e33c78bfa178bc21404f182c9fc64d
[root@host ~]# docker ps | grep dockerbug
# (no response, this container is not running)
[root@host ~]# docker start dockerbug
dockerbug

[jstaph]

Possibly related issues:
when docker daemon stop illegality, docker start container_id may failed #8065
docker fails to mount the block device for the container on devicemapper #4036

[lsm5]

@maxamillion @jperrin ... could you check if this is good for merge here, and also for the el6 rpm?

[jperrin]

Yeah, I'm okay with this for now. It's a valid point, although I think there might be a better way to solve this long term. This is a good short term fix.

[lsm5]

@jperrin cool thanks, I'll update the rpm with this.

@tianon ping

[smerrill]

It is an interesting problem, because the docker daemon will obviously also try to shut down all running containers when it gets the signal to shut down, and it will probably always know better than a shell script how to do that properly, especially since parts of the daemon can register additional tasks with eng.onShutdown().

I had originally thought of putting something in the init script that would loop through $(docker ps -q) and try to stop all containers individually, but I decided not to since that would be duplicating functionality that already exists in the daemon itself.

[tianon]

Seems hacky, but LGTM. You're good for a merge on this then, @lsm5?

[lsm5]

@tianon yup