restart container after reboot

[barrachri]

Hi people,
I'm trying docker-compose 1.10-rc1 on Fedora 21.

python:
  restart: always
  build: ./backend/
  environment:
   - APP_CONFIG=config.Production
nodejs:
  restart: always
  build: ./node_app/
nginx:
    restart: always
    build: ./deploy/nginx/
    ports:
        - "80:80"
        - "443:443"
    links:
        - python:pythonapp
        - nodejs:nodejs

But after a simple # reboot containers don't start again.

Maybe I miss something ?

[thaJeztah]

What version of Docker are you running? Perhaps add the output of docker version and docker -D info

[barrachri]

It was a Digital Ocean's droplet.
Docker version : 1.4.0-1.fc21

I'll make another try tomorrow.

[thaJeztah]

Thanks for adding that info; some distros still have older versions of Docker (pre 1.2) and the restart-policies were introduced with Docker 1.2 :)

[albers]

I can reproduce this with Compose 1.1.0-rc1 and Docker 1.4.1 on Ubuntu 14.04 under certain conditions that I do not understand yet:

Using a real world example, the containers will come up after restarting the docker service, but they will not come up after rebooting the host. After a reboot, the manually started services lost their restarting characteristics as well:

$ cat docker-compose.yml
db2:
  image: db2
  privileged: true
  ports:
    - "10.0.81.0:50000:50000"
  restart: always
kisserver:
  image: kisserver
  links:
    - db2
  restart: always
services:
  image: services
  links:
    - db2:minikis
    - kisserver
  ports:
    - "10.0.81.0:80:8080"
  restart: always

$ docker-compose up -d
Creating staging0_db2_1...
Creating staging0_kisserver_1...
Creating staging0_services_1...
$ docker-compose ps
        Name                      Command               State             Ports
------------------------------------------------------------------------------------------
staging0_db2_1         db2-init                         Up      10.0.81.0:50000->50000/tcp
staging0_kisserver_1   /bin/sh -c java -jar org.e ...   Up      8090/tcp
staging0_services_1    /opt/jboss/jboss/bin/stand ...   Up      10.0.81.0:80->8080/tcp

# restarting docker daemon is OK
$ sudo service docker restart
$ docker-compose ps
        Name                      Command               State             Ports
------------------------------------------------------------------------------------------
staging0_db2_1         db2-init                         Up      10.0.81.0:50000->50000/tcp
staging0_kisserver_1   /bin/sh -c java -jar org.e ...   Up      8090/tcp
staging0_services_1    /opt/jboss/jboss/bin/stand ...   Up      10.0.81.0:80->8080/tcp

# after rebooting, services do not come up
$ sudo reboot
[...]
$ docker-compose ps
        Name                      Command                State    Ports
-----------------------------------------------------------------------
staging0_db2_1         db2-init                         Exit -1
staging0_kisserver_1   /bin/sh -c java -jar org.e ...   Exit -1
staging0_services_1    /opt/jboss/jboss/bin/stand ...   Exit -1

# so we start them again.
$ docker-compose up -d
Recreating staging0_db2_1...
Recreating staging0_kisserver_1...
Recreating staging0_services_1...
$ docker-compose ps
        Name                      Command               State             Ports
------------------------------------------------------------------------------------------
staging0_db2_1         db2-init                         Up      10.0.81.0:50000->50000/tcp
staging0_kisserver_1   /bin/sh -c java -jar org.e ...   Up      8090/tcp
staging0_services_1    /opt/jboss/jboss/bin/stand ...   Up      10.0.81.0:80->8080/tcp

# but now our services do no longer come up after restarting docker daemon
$ sudo service docker restart
$ docker-compose ps
        Name                      Command                State     Ports
------------------------------------------------------------------------
staging0_db2_1         db2-init                         Exit 128
staging0_kisserver_1   /bin/sh -c java -jar org.e ...   Exit 128
staging0_services_1    /opt/jboss/jboss/bin/stand ...   Exit 128

But this behavior depends on service composition.
With this minimal "stack", nginx comes up under all conditions:

nginx:
    image: nginx
    restart: always

[thaJeztah]

@albers wondering if you docker inspect -f "{{ .HostConfig.RestartPolicy }}" <container>;

is the restart-policy stripped from the containers after restart?
and after the "so we start them again" step?

(Sorry, don't have a host at hand here myself that I can restart)

[aanand]

Can you reproduce this without Compose? i.e. with docker run --restart=always ...

[albers]

@thaJeztah At all stages, the output of docker inspect is:

$ docker inspect -f "{{ .HostConfig.RestartPolicy }}" staging0_db2_1 staging0_kisserver_1 staging0_services_1
map[MaximumRetryCount:0 Name:always]
map[MaximumRetryCount:0 Name:always]
map[MaximumRetryCount:0 Name:always]

[thaJeztah]

@albers thanks, I was wondering if Fig/Compose would strip the RestartPolicy somehow during the re-create steps.

[albers]

@aanand I manually created the application stack like this:

$ docker run -d --name staging0_db2_1 --privileged --publish 10.0.81.0:50000:50000 --restart always db2
$ docker run -d --name staging0_kisserver_1 --link staging0_db2_1:db2 --restart always kisserver
$ docker run -d --name staging0_services_1 --link staging0_db2_1:minikis --link staging0_kisserver_1:kisserver --publish 10.0.81.0:80:8080 --restart always services
$ docker-compose ps
        Name                      Command               State             Ports
------------------------------------------------------------------------------------------
staging0_db2_1         db2-init                         Up      10.0.81.0:50000->50000/tcp
staging0_kisserver_1   /bin/sh -c java -jar org.e ...   Up      8090/tcp
staging0_services_1    /opt/jboss/jboss/bin/stand ...   Up      10.0.81.0:80->8080/tcp

My findings:

• After restarting the docker daemon, the containers came up.
• After reboot, they did not.
• After manually restarting them with docker start and then restarting the docker daemon, they came up. This differs from the corresponding Compose szenario.

Looks like a Docker issue.

[wernight]

Dokku presentation video around minute 7 has a good way of handling zero-downtime deployment

[demarant]

it works for me on CentOS 7
Docker version 1.5.0, build a8a31ef
docker-compose 1.1.0

my docker-compose.yml:

web:
    restart: always
    image: nginx
    ports:
        - "80:80"
    volumes:
        - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
        - ./html:/usr/share/nginx/html:ro
    links:
        - cadvisor
cadvisor:
    restart: always
    image: google/cadvisor:latest
    volumes:
        - /:/rootfs:ro
        - /var/run:/var/run:rw
        - /sys:/sys:ro
        - /var/lib/docker/:/var/lib/docker:ro

I rebooted the VM in openstack and after the services where started up correctly:

[root@swarm01 ~]# docker ps
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS              PORTS                         NAMES
894fadbf5591        nginx:latest             "nginx -g 'daemon of   6 minutes ago       Up 4 minutes        443/tcp, 0.0.0.0:80->80/tcp   web_web_1
42eacd5cb050        google/cadvisor:0.10.1   "/usr/bin/cadvisor"    6 minutes ago       Up 4 minutes        8080/tcp                      web_cadvisor_1

I than killed nginx process manually, and it was restarted automatically. nice.

[root@swarm01 ~]# ps aux | grep nginx
root      1971  0.0  0.0  30964  2948 ?        Ss   11:52   0:00 nginx: master process nginx -g daemon off;
101       2004  0.0  0.0  31940  2588 ?        S    11:52   0:00 nginx: worker process
root      2041  0.0  0.0 112640   976 pts/0    S+   11:56   0:00 grep --color=auto nginx
[root@swarm01 ~]# kill 1971
[root@swarm01 ~]# ps aux | grep nginx
root      2063  1.0  0.0  30964  2944 ?        Ss   11:57   0:00 nginx: master process nginx -g daemon off;
101       2073  0.0  0.0  31940  2520 ?        S    11:57   0:00 nginx: worker process
root      2093  0.0  0.0 112640   972 pts/0    S+   11:57   0:00 grep --color=auto nginx
[root@swarm01 ~]# docker ps
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS              PORTS                         NAMES
894fadbf5591        nginx:latest             "nginx -g 'daemon of   7 minutes ago       Up 9 seconds        443/tcp, 0.0.0.0:80->80/tcp   web_web_1
42eacd5cb050        google/cadvisor:0.10.1   "/usr/bin/cadvisor"    7 minutes ago       Up 5 minutes        8080/tcp                      web_cadvisor_1

On the other hand If I use the docker CLI to stop nginx, it will not be restarted automatically. Looks logic to me, because of course using the docker CLI is an intentional stop.

[root@swarm01 ~]# docker stop 894fadbf5591
894fadbf5591
[root@swarm01 ~]# docker ps
CONTAINER ID        IMAGE                    COMMAND               CREATED             STATUS              PORTS               NAMES
42eacd5cb050        google/cadvisor:0.10.1   "/usr/bin/cadvisor"   16 minutes ago      Up 13 minutes       8080/tcp            web_cadvisor_1

the same apply if we use docker-compose CLI commands, if we stop a service intentionally, it will not be autorestarted.

[root@swarm01 web]# docker-compose ps
     Name              Command          State              Ports
---------------------------------------------------------------------------
web_cadvisor_1   /usr/bin/cadvisor      Up      8080/tcp
web_web_1        nginx -g daemon off;   Up      443/tcp, 0.0.0.0:80->80/tcp
[root@swarm01 web]# docker inspect -f "{{ .HostConfig.RestartPolicy }}" web_web_1
map[MaximumRetryCount:0 Name:always]
[root@swarm01 web]# docker-compose stop web
Stopping web_web_1...
[root@swarm01 web]# docker-compose ps
     Name              Command          State     Ports
---------------------------------------------------------
web_cadvisor_1   /usr/bin/cadvisor      Up       8080/tcp
web_web_1        nginx -g daemon off;   Exit 0

So far so good in my case scenario.

[peter-leonov]

+1

The same Exited (128) for a restart: always container.

DigitalOcean droplet created with Docker Machine version 0.2.0 (HEAD), container created with Docker Compose version: 1.3.0 on Docker 1.6.2.

docker inspect -f "{{ .HostConfig.RestartPolicy }}" containerId

map[Name:always MaximumRetryCount:0]

I should add that Upstart kept this daemon up for years on the came DigitalOcean/Ubuntu/Rails before I switched to Docker.

[trukhinyuri]

+1
The same Exited (128) for a restart: always container.

Docker version 1.6.1, build a8a31ef/1.6.1
docker-compose version: 1.3.1
map[MaximumRetryCount:0 Name:always]
OS: Oracle Linux 7.1

After reboot docker start failed:
docker start 4944630b36fd
Error response from daemon: Cannot start container 4944630b36fd: Error getting container 4944630b36fdf2f0f23d4406a9880f5a4c73f3db12ea30cacc1291a563b4225b from driver devicemapper: Error mounting '/dev/mapper/docker-8:1-524343-4944630b36fdf2f0f23d4406a9880f5a4c73f3db12ea30cacc1291a563b4225b' on '/var/lib/docker/devicemapper/mnt/4944630b36fdf2f0f23d4406a9880f5a4c73f3db12ea30cacc1291a563b4225b': invalid argument
FATA[0000] Error: failed to start one or more containers

[dnephin]

I think this is ultimately a docker engine bug. After compose starts the containers it's up to engine to restart them.

[thaJeztah]

@dnephin agreed; also, All Docker versions reported here are now getting old; this issue could be resolved now, but hard to tell without, e.g. daemon logs

[passcod]

Works for me here with Docker 1.7.1, Compose 1.4.0, on CoreOS 766.3.0.

[dnephin]

Sounds like this was not a compose issue, and has been fixed in engine

[da3dsoul]

I'm having this exact problem with Docker on Windows. I'm using nextcloud with docker-compose. If I docker-compose stop then docker-compose up -d, then it works, but after a Windows reboot, docker starts, the containers start, but apache/nginx can't open the port. stop and up -d will then fix the problem again. It shouldn't be necessary to run a timed task after startup to restart the containers.

[aaaguirrep]

Hi, I am having the same problem with ubuntu 19.04.

[nerdybynature]

Issue also ocurring on Windows Server 2019: Reboot of host does not start up containers.

[kustrun]

Issue with restart: always also present on Ubuntu 19.04 with Docker version 18.09.7, build 2d0083d. Any possible solution?

[AmaHacka]

@kustrun I got same trouble with Docker version 18.09.7 on Ubuntu 18.04. But I solved this problem with sudo systemctl enable docker. I just missed to enable docker service autostart...

[djarvis]

Issue also ocurring on Windows Server 2019: Reboot of host does not start up containers.

Same here. Windows Server 2019. Rebooting machine does not start up containers. ps -a shows containers still there but stopped.

[shakeelosmani]

Same issues here. Windows server 2019. I am not sure why issue is closed

[shakeelosmani]

Update for me I had to do couple of things, switch the docker-compose file to 2.2 so that restart will work on compose mode. Secondly I realized when machines when host is rebooted, the old network deleted but the the containers try to use that and fail. The right way to do it was define a network and let it get created then machines come back online no issues

[djarvis]

@shakeelosmani Tried what you said but still my containers refuse to start. I define a network and have the containers use it, everything starts up and shuts down OK but stay in "exited" on a reboot. This is on Docker for Windows which I think has some issues with this.