New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FIXED] Docker stopped working after 'yum update' and reboot (Centos 7) #12087

Closed
ctmnz opened this Issue Apr 4, 2015 · 33 comments

Comments

Projects
None yet
@ctmnz

ctmnz commented Apr 4, 2015

[root@cos7 data]# uname -a
Linux cos7 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@cos7 data]# docker -v
Docker version 1.5.0-dev, build fc0329b/1.5.0
[root@cos7 data]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[root@cos7 data]#
[root@cos7 data]# systemctl start docker
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.
[root@cos7 data]# systemctl status docker.service
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: failed (Result: exit-code) since Sun 2015-04-05 02:41:58 EEST; 5s ago
     Docs: http://docs.docker.com
  Process: 14431 ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
 Main PID: 14431 (code=exited, status=1/FAILURE)

Apr 05 02:41:58 cos7 docker[14431]: time="2015-04-05T02:41:58+03:00" level="info" msg="+job serveapi(unix:///var/run/docker.sock)"
Apr 05 02:41:58 cos7 docker[14431]: time="2015-04-05T02:41:58+03:00" level="fatal" msg="EOF"
Apr 05 02:41:58 cos7 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 05 02:41:58 cos7 systemd[1]: Failed to start Docker Application Container Engine.
Apr 05 02:41:58 cos7 systemd[1]: Unit docker.service entered failed state.
[root@cos7 data]#

@ctmnz ctmnz changed the title from Docker stopped to work after reboot (Centos 7) to Docker stopped working after reboot (Centos 7) Apr 4, 2015

@ctmnz ctmnz changed the title from Docker stopped working after reboot (Centos 7) to Docker stopped working after 'yum update' and reboot (Centos 7) Apr 4, 2015

@ctmnz

This comment has been minimized.

Show comment
Hide comment
@ctmnz

ctmnz Apr 5, 2015

I just saw that I don't see the docker virtual NIC 'docker0' in ifconfig anymore.

ctmnz commented Apr 5, 2015

I just saw that I don't see the docker virtual NIC 'docker0' in ifconfig anymore.

@phemmer

This comment has been minimized.

Show comment
Hide comment
@phemmer

phemmer Apr 5, 2015

Contributor

This line doesn't smell right:

Apr 05 02:41:58 cos7 docker[14431]: time="2015-04-05T02:41:58+03:00" level="info" msg="+job serveapi(unix:///var/run/docker.sock)"

This looks like the docker daemon is either being started as docker -d -H unix:///var/run/unix.sock or as docker -d (with no -H).
You're using systemd, which means docker should be using -H fd://, and be completely unaware of /var/run/unix.sock.

Do you have an /etc/sysconfig/docker? Can you provide the contents?
Can you also provide systemctl status docker.socket?

Contributor

phemmer commented Apr 5, 2015

This line doesn't smell right:

Apr 05 02:41:58 cos7 docker[14431]: time="2015-04-05T02:41:58+03:00" level="info" msg="+job serveapi(unix:///var/run/docker.sock)"

This looks like the docker daemon is either being started as docker -d -H unix:///var/run/unix.sock or as docker -d (with no -H).
You're using systemd, which means docker should be using -H fd://, and be completely unaware of /var/run/unix.sock.

Do you have an /etc/sysconfig/docker? Can you provide the contents?
Can you also provide systemctl status docker.socket?

@ctmnz

This comment has been minimized.

Show comment
Hide comment
@ctmnz

ctmnz Apr 5, 2015

@phemmer . Hello.
I have provided systemctl status docker.socket (it is the last command that I have shown the result of).
I have "OPTIONS=-H fd://" in my /etc/sysconfig/docker

ctmnz commented Apr 5, 2015

@phemmer . Hello.
I have provided systemctl status docker.socket (it is the last command that I have shown the result of).
I have "OPTIONS=-H fd://" in my /etc/sysconfig/docker

@lazabogdan

This comment has been minimized.

Show comment
Hide comment
@lazabogdan

lazabogdan Apr 6, 2015

Got the same issue.

Did "sudo yum install device-mapper-event-libs" and then "sudo yum reinstall docker" and problem solved.

lazabogdan commented Apr 6, 2015

Got the same issue.

Did "sudo yum install device-mapper-event-libs" and then "sudo yum reinstall docker" and problem solved.

@ctmnz

This comment has been minimized.

Show comment
Hide comment
@ctmnz

ctmnz Apr 6, 2015

Nope... it doesn't start.

[root@cos7 ~]# sudo yum install device-mapper-event-libs
Loaded plugins: fastestmirror, langpacks
base                                                                                                                                                                    | 3.6 kB  00:00:00     
epel/x86_64/metalink                                                                                                                                                    |  27 kB  00:00:00     
epel                                                                                                                                                                    | 4.4 kB  00:00:00     
extras                                                                                                                                                                  | 3.4 kB  00:00:00     
updates                                                                                                                                                                 | 3.4 kB  00:00:00     
Loading mirror speeds from cached hostfile
 * base: mirrors.neterra.net
 * epel: mirrors.neterra.net
 * extras: mirrors.neterra.net
 * updates: mirrors.neterra.net
Package 7:device-mapper-event-libs-1.02.93-3.el7.x86_64 already installed and latest version
Nothing to do
[root@cos7 ~]# sudo yum reinstall docker
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.neterra.net
 * epel: mirrors.neterra.net
 * extras: mirrors.neterra.net
 * updates: mirrors.neterra.net
Resolving Dependencies
--> Running transaction check
---> Package docker.x86_64 0:1.5.0-28.el7.centos will be reinstalled
--> Finished Dependency Resolution
Dependencies Resolved
===============================================================================================================================================================================================
 Package                                   Arch                                      Version                                                   Repository                                 Size
===============================================================================================================================================================================================
Reinstalling:
 docker                                    x86_64                                    1.5.0-28.el7.centos                                       extras                                    7.2 M
Transaction Summary
===============================================================================================================================================================================================
Reinstall  1 Package
Total download size: 7.2 M
Installed size: 31 M
Is this ok [y/d/N]: y
Downloading packages:
docker-1.5.0-28.el7.centos.x86_64.rpm                                                                                                                                   | 7.2 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : docker-1.5.0-28.el7.centos.x86_64                                                                                                                                           1/1 
  Verifying  : docker-1.5.0-28.el7.centos.x86_64                                                                                                                                           1/1 
Installed:
  docker.x86_64 0:1.5.0-28.el7.centos                                                                                                                                                          
Complete!
[root@cos7 ~]# systemctl start docker
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.
[root@cos7 ~]# systemctl status docker.service 
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: failed (Result: exit-code) since Mon 2015-04-06 11:16:20 EEST; 17s ago
     Docs: http://docs.docker.com
  Process: 15889 ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
 Main PID: 15889 (code=exited, status=1/FAILURE)
Apr 06 11:16:20 cos7 docker[15889]: time="2015-04-06T11:16:20+03:00" level="info" msg="+job serveapi(fd://)"
Apr 06 11:16:20 cos7 docker[15889]: time="2015-04-06T11:16:20+03:00" level="info" msg="Listening for HTTP on fd ()"
Apr 06 11:16:20 cos7 docker[15889]: No sockets found
Apr 06 11:16:20 cos7 docker[15889]: time="2015-04-06T11:16:20+03:00" level="info" msg="-job serveapi(fd://) = ERR (1)"
Apr 06 11:16:20 cos7 docker[15889]: time="2015-04-06T11:16:20+03:00" level="fatal" msg="No sockets found"
Apr 06 11:16:20 cos7 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 06 11:16:20 cos7 systemd[1]: Failed to start Docker Application Container Engine.
Apr 06 11:16:20 cos7 systemd[1]: Unit docker.service entered failed state.

ctmnz commented Apr 6, 2015

Nope... it doesn't start.

[root@cos7 ~]# sudo yum install device-mapper-event-libs
Loaded plugins: fastestmirror, langpacks
base                                                                                                                                                                    | 3.6 kB  00:00:00     
epel/x86_64/metalink                                                                                                                                                    |  27 kB  00:00:00     
epel                                                                                                                                                                    | 4.4 kB  00:00:00     
extras                                                                                                                                                                  | 3.4 kB  00:00:00     
updates                                                                                                                                                                 | 3.4 kB  00:00:00     
Loading mirror speeds from cached hostfile
 * base: mirrors.neterra.net
 * epel: mirrors.neterra.net
 * extras: mirrors.neterra.net
 * updates: mirrors.neterra.net
Package 7:device-mapper-event-libs-1.02.93-3.el7.x86_64 already installed and latest version
Nothing to do
[root@cos7 ~]# sudo yum reinstall docker
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.neterra.net
 * epel: mirrors.neterra.net
 * extras: mirrors.neterra.net
 * updates: mirrors.neterra.net
Resolving Dependencies
--> Running transaction check
---> Package docker.x86_64 0:1.5.0-28.el7.centos will be reinstalled
--> Finished Dependency Resolution
Dependencies Resolved
===============================================================================================================================================================================================
 Package                                   Arch                                      Version                                                   Repository                                 Size
===============================================================================================================================================================================================
Reinstalling:
 docker                                    x86_64                                    1.5.0-28.el7.centos                                       extras                                    7.2 M
Transaction Summary
===============================================================================================================================================================================================
Reinstall  1 Package
Total download size: 7.2 M
Installed size: 31 M
Is this ok [y/d/N]: y
Downloading packages:
docker-1.5.0-28.el7.centos.x86_64.rpm                                                                                                                                   | 7.2 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : docker-1.5.0-28.el7.centos.x86_64                                                                                                                                           1/1 
  Verifying  : docker-1.5.0-28.el7.centos.x86_64                                                                                                                                           1/1 
Installed:
  docker.x86_64 0:1.5.0-28.el7.centos                                                                                                                                                          
Complete!
[root@cos7 ~]# systemctl start docker
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.
[root@cos7 ~]# systemctl status docker.service 
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: failed (Result: exit-code) since Mon 2015-04-06 11:16:20 EEST; 17s ago
     Docs: http://docs.docker.com
  Process: 15889 ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
 Main PID: 15889 (code=exited, status=1/FAILURE)
Apr 06 11:16:20 cos7 docker[15889]: time="2015-04-06T11:16:20+03:00" level="info" msg="+job serveapi(fd://)"
Apr 06 11:16:20 cos7 docker[15889]: time="2015-04-06T11:16:20+03:00" level="info" msg="Listening for HTTP on fd ()"
Apr 06 11:16:20 cos7 docker[15889]: No sockets found
Apr 06 11:16:20 cos7 docker[15889]: time="2015-04-06T11:16:20+03:00" level="info" msg="-job serveapi(fd://) = ERR (1)"
Apr 06 11:16:20 cos7 docker[15889]: time="2015-04-06T11:16:20+03:00" level="fatal" msg="No sockets found"
Apr 06 11:16:20 cos7 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 06 11:16:20 cos7 systemd[1]: Failed to start Docker Application Container Engine.
Apr 06 11:16:20 cos7 systemd[1]: Unit docker.service entered failed state.
@ctmnz

This comment has been minimized.

Show comment
Hide comment
@ctmnz

ctmnz Apr 6, 2015

It looks like there is a problem with my /var/lib/docker directory because when I renamed it on 'docker-broken' and tried to start docker with 'systemctl start docker' the problem disappeared (but my images and containers also). How can I check what is wrong with my /var/lib/docker directory? I have not operated via console commands (copy/rm/mv files) in this directory.

ctmnz commented Apr 6, 2015

It looks like there is a problem with my /var/lib/docker directory because when I renamed it on 'docker-broken' and tried to start docker with 'systemctl start docker' the problem disappeared (but my images and containers also). How can I check what is wrong with my /var/lib/docker directory? I have not operated via console commands (copy/rm/mv files) in this directory.

@robbyt

This comment has been minimized.

Show comment
Hide comment
@robbyt

robbyt Apr 6, 2015

Contributor

This is possibly related to #12108

Contributor

robbyt commented Apr 6, 2015

This is possibly related to #12108

@ehazlett

This comment has been minimized.

Show comment
Hide comment
@ehazlett

ehazlett Apr 6, 2015

Contributor

thanks @robbyt

@ctmnz could you try a full system update to see if that helps?

Contributor

ehazlett commented Apr 6, 2015

thanks @robbyt

@ctmnz could you try a full system update to see if that helps?

@ctmnz

This comment has been minimized.

Show comment
Hide comment
@ctmnz

ctmnz Apr 6, 2015

@ehazlett The problem appeared after full system update. I am afraid that I won't be able to update my operation system more than the current state.

@robryk it is not from my device mapper. As i said in my previous comment, when I rename/delete /var/lib/docker my docker is starting fine (without containers and images). The problem is not in some driver or library. It is in this '/var/lib/docker' directory, but I don't know how to debug it more.

ctmnz commented Apr 6, 2015

@ehazlett The problem appeared after full system update. I am afraid that I won't be able to update my operation system more than the current state.

@robryk it is not from my device mapper. As i said in my previous comment, when I rename/delete /var/lib/docker my docker is starting fine (without containers and images). The problem is not in some driver or library. It is in this '/var/lib/docker' directory, but I don't know how to debug it more.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Apr 6, 2015

Contributor

@ctmnz Sounds like you have a hard to kill mount sitting @ /var/lib/docker or something.
Can you check /proc/mounts while docker is stopped?

Contributor

cpuguy83 commented Apr 6, 2015

@ctmnz Sounds like you have a hard to kill mount sitting @ /var/lib/docker or something.
Can you check /proc/mounts while docker is stopped?

@ctmnz

This comment has been minimized.

Show comment
Hide comment
@ctmnz

ctmnz Apr 7, 2015

ping @cpuguy83
Checked.

[root@cos7 lib]# systemctl stop docker
[root@cos7 lib]# mv docker docker-working/
[root@cos7 lib]# mv docker-broken/ docker
[root@cos7 lib]# systemctl start docker
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.
[root@cos7 lib]# systemctl status docker
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled)
   Active: failed (Result: exit-code) since Tue 2015-04-07 09:38:48 EEST; 4s ago
     Docs: http://docs.docker.com
  Process: 16989 ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
 Main PID: 16989 (code=exited, status=1/FAILURE)
Apr 07 09:38:48 cos7 docker[16989]: time="2015-04-07T09:38:48+03:00" level="info" msg="+job serveapi(unix:///var/run/....sock)"
Apr 07 09:38:48 cos7 docker[16989]: time="2015-04-07T09:38:48+03:00" level="fatal" msg="EOF"
Apr 07 09:38:48 cos7 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 07 09:38:48 cos7 systemd[1]: Failed to start Docker Application Container Engine.
Apr 07 09:38:48 cos7 systemd[1]: Unit docker.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.
[root@cos7 lib]# cat /proc/mounts 
rootfs / rootfs rw 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=3996496k,nr_inodes=999124,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/sda1 / xfs rw,relatime,attr2,inode64,noquota 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
/dev/sdb2 /data xfs rw,relatime,attr2,inode64,noquota 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
[root@cos7 lib]# 

I also checked this:

[root@cos7 lib]# cat /proc/mounts > /root/broken.txt
[root@cos7 lib]# mv docker docker-broken
[root@cos7 lib]# mv docker-working/ docker
[root@cos7 lib]# systemctl start docker
[root@cos7 lib]# cat /proc/mounts > /root/working.txt
[root@cos7 lib]# diff /root/broken.txt /root/working.txt 
[root@cos7 lib]# 

ctmnz commented Apr 7, 2015

ping @cpuguy83
Checked.

[root@cos7 lib]# systemctl stop docker
[root@cos7 lib]# mv docker docker-working/
[root@cos7 lib]# mv docker-broken/ docker
[root@cos7 lib]# systemctl start docker
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.
[root@cos7 lib]# systemctl status docker
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled)
   Active: failed (Result: exit-code) since Tue 2015-04-07 09:38:48 EEST; 4s ago
     Docs: http://docs.docker.com
  Process: 16989 ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
 Main PID: 16989 (code=exited, status=1/FAILURE)
Apr 07 09:38:48 cos7 docker[16989]: time="2015-04-07T09:38:48+03:00" level="info" msg="+job serveapi(unix:///var/run/....sock)"
Apr 07 09:38:48 cos7 docker[16989]: time="2015-04-07T09:38:48+03:00" level="fatal" msg="EOF"
Apr 07 09:38:48 cos7 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 07 09:38:48 cos7 systemd[1]: Failed to start Docker Application Container Engine.
Apr 07 09:38:48 cos7 systemd[1]: Unit docker.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.
[root@cos7 lib]# cat /proc/mounts 
rootfs / rootfs rw 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=3996496k,nr_inodes=999124,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/sda1 / xfs rw,relatime,attr2,inode64,noquota 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
/dev/sdb2 /data xfs rw,relatime,attr2,inode64,noquota 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
[root@cos7 lib]# 

I also checked this:

[root@cos7 lib]# cat /proc/mounts > /root/broken.txt
[root@cos7 lib]# mv docker docker-broken
[root@cos7 lib]# mv docker-working/ docker
[root@cos7 lib]# systemctl start docker
[root@cos7 lib]# cat /proc/mounts > /root/working.txt
[root@cos7 lib]# diff /root/broken.txt /root/working.txt 
[root@cos7 lib]# 
@jmccormick2001

This comment has been minimized.

Show comment
Hide comment
@jmccormick2001

jmccormick2001 Apr 9, 2015

I had the same problem, removed docker, then manually removed /var/lib/docker, then re-installed docker and it appears to work now.

However, I found something new today, if I add '-H fd://" to /etc/sysconfig/docker, then I get the "no sockets found" error during startup again.

jmccormick2001 commented Apr 9, 2015

I had the same problem, removed docker, then manually removed /var/lib/docker, then re-installed docker and it appears to work now.

However, I found something new today, if I add '-H fd://" to /etc/sysconfig/docker, then I get the "no sockets found" error during startup again.

@ctmnz

This comment has been minimized.

Show comment
Hide comment
@ctmnz

ctmnz Apr 14, 2015

Additional information after turning on -D (debug mode)

-- Logs begin at Sat 2015-04-04 23:51:54 EEST, end at Tue 2015-04-14 15:02:27 EEST. --
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm version   OF   [16384] (*1)"
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="devicemapper: udev sync support: true"
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="Generated prefix: docker-8:1-1612350994"
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="Checking for existence of the pool 'docker-8:1-1612350994-pool'"
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm info docker-8:1-1612350994-pool  OF  
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm status docker-8:1-1612350994-pool  OF
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="fatal" msg="EOF"
Apr 14 15:02:27 cos7 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 14 15:02:27 cos7 systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has failed.
-- 
-- The result is failed.
Apr 14 15:02:27 cos7 systemd[1]: Unit docker.service entered failed state.

ctmnz commented Apr 14, 2015

Additional information after turning on -D (debug mode)

-- Logs begin at Sat 2015-04-04 23:51:54 EEST, end at Tue 2015-04-14 15:02:27 EEST. --
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm version   OF   [16384] (*1)"
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="devicemapper: udev sync support: true"
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="Generated prefix: docker-8:1-1612350994"
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="Checking for existence of the pool 'docker-8:1-1612350994-pool'"
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm info docker-8:1-1612350994-pool  OF  
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="debug" msg="libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm status docker-8:1-1612350994-pool  OF
Apr 14 15:02:27 cos7 docker[20005]: time="2015-04-14T15:02:27+03:00" level="fatal" msg="EOF"
Apr 14 15:02:27 cos7 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 14 15:02:27 cos7 systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has failed.
-- 
-- The result is failed.
Apr 14 15:02:27 cos7 systemd[1]: Unit docker.service entered failed state.
@ctmnz

This comment has been minimized.

Show comment
Hide comment
@ctmnz

ctmnz Apr 14, 2015

strace information:

munmap(0x7f01da166000, 4096)            = 0
ioctl(4, DM_VERSION, 0x29c56f0)         = 0
ioctl(4, DM_VERSION, 0x29c5620)         = 0
semctl(0, 0, SEM_INFO, 0x7fff487267b0)  = 6
open("/etc/udev/udev.conf", O_RDONLY|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=44, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f01da166000
read(5, "# see udev(7) for details\n\n#udev"..., 4096) = 44
read(5, "", 4096)                       = 0
close(5)                                = 0
munmap(0x7f01da166000, 4096)            = 0
open("/run/udev/queue.bin", O_RDONLY|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=8, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f01da166000
read(5, "|G\0\0\0\0\0\0", 4096)         = 8
close(5)                                = 0
munmap(0x7f01da166000, 4096)            = 0
stat("/var/lib/docker/devicemapper/metadata", {st_mode=S_IFDIR|0700, st_size=49152, ...}) = 0
stat("/var/lib/docker/devicemapper", {st_mode=S_IFDIR|0700, st_size=50, ...}) = 0
ioctl(4, DM_DEV_STATUS, 0x29c97b0)      = 0
open("/proc/self/fd", O_RDONLY|O_CLOEXEC) = 5
getdents64(5, /* 8 entries */, 4096)    = 192
getdents64(5, /* 0 entries */, 4096)    = 0
lstat("/proc/self/fd/0", {st_mode=S_IFLNK|0700, st_size=64, ...}) = 0
lstat("/proc/self/fd/1", {st_mode=S_IFLNK|0700, st_size=64, ...}) = 0
lstat("/proc/self/fd/2", {st_mode=S_IFLNK|0700, st_size=64, ...}) = 0
lstat("/proc/self/fd/3", {st_mode=S_IFLNK|0500, st_size=64, ...}) = 0
lstat("/proc/self/fd/4", {st_mode=S_IFLNK|0700, st_size=64, ...}) = 0
lstat("/proc/self/fd/5", {st_mode=S_IFLNK|0500, st_size=64, ...}) = 0
close(5)                                = 0
readlink("/proc/self/fd/0", "/dev/pts/1", 128) = 10
readlink("/proc/self/fd/1", "/dev/pts/1", 128) = 10
readlink("/proc/self/fd/2", "/dev/pts/1", 128) = 10
readlink("/proc/self/fd/3", "/dev/urandom", 128) = 12
readlink("/proc/self/fd/4", "/dev/mapper/control", 128) = 19
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
readlink("/proc/self/fd/5", 0xc208062980, 128) = -1 ENOENT (No such file or directory)
open("/var/lib/docker/devicemapper/devicemapper/json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
ioctl(4, DM_TABLE_STATUS, 0x29cd890)    = 0
write(2, "\33[31mFATA\33[0m[0000] EOF         "..., 66FATA[0000] EOF                                          
) = 66
exit_group(1)                           = ?
+++ exited with 1 +++

ctmnz commented Apr 14, 2015

strace information:

munmap(0x7f01da166000, 4096)            = 0
ioctl(4, DM_VERSION, 0x29c56f0)         = 0
ioctl(4, DM_VERSION, 0x29c5620)         = 0
semctl(0, 0, SEM_INFO, 0x7fff487267b0)  = 6
open("/etc/udev/udev.conf", O_RDONLY|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=44, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f01da166000
read(5, "# see udev(7) for details\n\n#udev"..., 4096) = 44
read(5, "", 4096)                       = 0
close(5)                                = 0
munmap(0x7f01da166000, 4096)            = 0
open("/run/udev/queue.bin", O_RDONLY|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=8, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f01da166000
read(5, "|G\0\0\0\0\0\0", 4096)         = 8
close(5)                                = 0
munmap(0x7f01da166000, 4096)            = 0
stat("/var/lib/docker/devicemapper/metadata", {st_mode=S_IFDIR|0700, st_size=49152, ...}) = 0
stat("/var/lib/docker/devicemapper", {st_mode=S_IFDIR|0700, st_size=50, ...}) = 0
ioctl(4, DM_DEV_STATUS, 0x29c97b0)      = 0
open("/proc/self/fd", O_RDONLY|O_CLOEXEC) = 5
getdents64(5, /* 8 entries */, 4096)    = 192
getdents64(5, /* 0 entries */, 4096)    = 0
lstat("/proc/self/fd/0", {st_mode=S_IFLNK|0700, st_size=64, ...}) = 0
lstat("/proc/self/fd/1", {st_mode=S_IFLNK|0700, st_size=64, ...}) = 0
lstat("/proc/self/fd/2", {st_mode=S_IFLNK|0700, st_size=64, ...}) = 0
lstat("/proc/self/fd/3", {st_mode=S_IFLNK|0500, st_size=64, ...}) = 0
lstat("/proc/self/fd/4", {st_mode=S_IFLNK|0700, st_size=64, ...}) = 0
lstat("/proc/self/fd/5", {st_mode=S_IFLNK|0500, st_size=64, ...}) = 0
close(5)                                = 0
readlink("/proc/self/fd/0", "/dev/pts/1", 128) = 10
readlink("/proc/self/fd/1", "/dev/pts/1", 128) = 10
readlink("/proc/self/fd/2", "/dev/pts/1", 128) = 10
readlink("/proc/self/fd/3", "/dev/urandom", 128) = 12
readlink("/proc/self/fd/4", "/dev/mapper/control", 128) = 19
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
readlink("/proc/self/fd/5", 0xc208062980, 128) = -1 ENOENT (No such file or directory)
open("/var/lib/docker/devicemapper/devicemapper/json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
ioctl(4, DM_TABLE_STATUS, 0x29cd890)    = 0
write(2, "\33[31mFATA\33[0m[0000] EOF         "..., 66FATA[0000] EOF                                          
) = 66
exit_group(1)                           = ?
+++ exited with 1 +++
@ctmnz

This comment has been minimized.

Show comment
Hide comment
@ctmnz

ctmnz Apr 14, 2015

device mapper info:

[root@cos7 lib]# dmsetup info docker-8:1-1612350994-pool
Name:              docker-8:1-1612350994-pool
State:             ACTIVE
Read Ahead:        256
Tables present:    INACTIVE
Open count:        0
Event number:      0
Major, minor:      253, 0
Number of targets: 0

ctmnz commented Apr 14, 2015

device mapper info:

[root@cos7 lib]# dmsetup info docker-8:1-1612350994-pool
Name:              docker-8:1-1612350994-pool
State:             ACTIVE
Read Ahead:        256
Tables present:    INACTIVE
Open count:        0
Event number:      0
Major, minor:      253, 0
Number of targets: 0
@ctmnz

This comment has been minimized.

Show comment
Hide comment
@ctmnz

ctmnz Apr 14, 2015

So.. I have fixed the problem manually. It was a problem with the device mapper

After clearing the volume with the command dmsetup clear docker-8:1-1612350994-pool and reboot the docker started without problem.

:

[root@cos7 lib]# dmsetup info docker-8:1-1612350994-pool
Name:              docker-8:1-1612350994-pool
State:             ACTIVE
Read Ahead:        256
Tables present:    INACTIVE
Open count:        0
Event number:      0
Major, minor:      253, 0
Number of targets: 0
[root@cos7 lib]# dmsetup clear docker-8:1-1612350994-pool
[root@cos7 lib]# 
[root@cos7 lib]# dmsetup info docker-8:1-1612350994-pool
Name:              docker-8:1-1612350994-pool
State:             ACTIVE
Read Ahead:        256
Tables present:    None
Open count:        0
Event number:      0
Major, minor:      253, 0
Number of targets: 0
[root@cos7 lib]# reboot 
[root@cos7 ~]# systemctl start docker
[root@cos7 ~]# 
[root@cos7 ~]# docker info 
Containers: 8
Images: 580
Storage Driver: devicemapper
 Pool Name: docker-8:1-1612350994-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 11.86 GB
 Data Space Total: 107.4 GB
 Data Space Available: 95.52 GB
 Metadata Space Used: 23.96 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.124 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.1.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 8
Total Memory: 7.639 GiB
Name: cos7
ID: NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE
Debug mode (server): true
Debug mode (client): false
Fds: 12
Goroutines: 14
EventsListeners: 0
Init SHA1: 444444444444444444431735b44444444444
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker

ctmnz commented Apr 14, 2015

So.. I have fixed the problem manually. It was a problem with the device mapper

After clearing the volume with the command dmsetup clear docker-8:1-1612350994-pool and reboot the docker started without problem.

:

[root@cos7 lib]# dmsetup info docker-8:1-1612350994-pool
Name:              docker-8:1-1612350994-pool
State:             ACTIVE
Read Ahead:        256
Tables present:    INACTIVE
Open count:        0
Event number:      0
Major, minor:      253, 0
Number of targets: 0
[root@cos7 lib]# dmsetup clear docker-8:1-1612350994-pool
[root@cos7 lib]# 
[root@cos7 lib]# dmsetup info docker-8:1-1612350994-pool
Name:              docker-8:1-1612350994-pool
State:             ACTIVE
Read Ahead:        256
Tables present:    None
Open count:        0
Event number:      0
Major, minor:      253, 0
Number of targets: 0
[root@cos7 lib]# reboot 
[root@cos7 ~]# systemctl start docker
[root@cos7 ~]# 
[root@cos7 ~]# docker info 
Containers: 8
Images: 580
Storage Driver: devicemapper
 Pool Name: docker-8:1-1612350994-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 11.86 GB
 Data Space Total: 107.4 GB
 Data Space Available: 95.52 GB
 Metadata Space Used: 23.96 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.124 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.1.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 8
Total Memory: 7.639 GiB
Name: cos7
ID: NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE:NOPE
Debug mode (server): true
Debug mode (client): false
Fds: 12
Goroutines: 14
EventsListeners: 0
Init SHA1: 444444444444444444431735b44444444444
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker

@ctmnz ctmnz changed the title from Docker stopped working after 'yum update' and reboot (Centos 7) to [FIXED] Docker stopped working after 'yum update' and reboot (Centos 7) Apr 14, 2015

@visualphoenix

This comment has been minimized.

Show comment
Hide comment
@visualphoenix

visualphoenix Jun 25, 2015

If this is fixed, maybe the issue should be closed?

visualphoenix commented Jun 25, 2015

If this is fixed, maybe the issue should be closed?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jul 13, 2015

Member

yup, closing

Member

thaJeztah commented Jul 13, 2015

yup, closing

@thaJeztah thaJeztah closed this Jul 13, 2015

@gmalouf

This comment has been minimized.

Show comment
Hide comment
@gmalouf

gmalouf Aug 18, 2015

Why is this being closed - it appears the dev had to do a manual step to resolve it. I'm getting the same issue and now have to run extra manual commands to get by this.

gmalouf commented Aug 18, 2015

Why is this being closed - it appears the dev had to do a manual step to resolve it. I'm getting the same issue and now have to run extra manual commands to get by this.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Aug 18, 2015

Member

@gmalouf it's being closed because, now the problem has been resolved for the OP, there's no reproducable case and nothing for us to debug. If you're having the same problem, please open a new issue (and a link to this one), so that we can start a fresh investigation into the problem.

Member

thaJeztah commented Aug 18, 2015

@gmalouf it's being closed because, now the problem has been resolved for the OP, there's no reproducable case and nothing for us to debug. If you're having the same problem, please open a new issue (and a link to this one), so that we can start a fresh investigation into the problem.

@vmadman

This comment has been minimized.

Show comment
Hide comment
@vmadman

vmadman Sep 1, 2015

Strange choice to close the issue, I think. Several people reported the same problem and its happening for me as well. I don't want to step on any toes, you guys are great, but its a strange approach to issue management, in my opinion. In the software I build.. I assume that any problem is the software's problem.. even if the OP is the only person who EVER sees this issue.. Docker should still detect it and report an error that properly explains what's happening.

I dunno, just seems strange to me...

vmadman commented Sep 1, 2015

Strange choice to close the issue, I think. Several people reported the same problem and its happening for me as well. I don't want to step on any toes, you guys are great, but its a strange approach to issue management, in my opinion. In the software I build.. I assume that any problem is the software's problem.. even if the OP is the only person who EVER sees this issue.. Docker should still detect it and report an error that properly explains what's happening.

I dunno, just seems strange to me...

@vmadman

This comment has been minimized.

Show comment
Hide comment
@vmadman

vmadman Sep 1, 2015

Aug 31 20:51:38 project.c2c.local systemd[1]: Starting Docker Application Container Engine...
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.516955606-04:00" level=error msg="WARNING: No --storage-opt dm.thinpooldev specified, using loopback; this configuration is strongly discouraged for production use"
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.539035205-04:00" level=error msg="ServeAPI error: No sockets found"
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.617245851-04:00" level=info msg="[graphdriver] using prior storage driver \"devicemapper\""
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.620607898-04:00" level=warning msg="Running modprobe bridge nf_nat br_netfilter failed with message: , error: exit status 1"
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.624593163-04:00" level=info msg="Firewalld running: false"
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.752114083-04:00" level=info msg="Loading containers: start."
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.752424509-04:00" level=info msg="Loading containers: done."
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.752445958-04:00" level=info msg="Daemon has completed initialization"
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.752465223-04:00" level=info msg="Docker daemon" commit="3043001/1.7.1" execdriver=native-0.2 graphdriver=devicemapper version=1.7.1
Aug 31 20:51:38 project.c2c.local systemd[1]: Started Docker Application Container Engine.
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.790898648-04:00" level=fatal msg="Shutting down due to ServeAPI error: No sockets found"
Aug 31 20:51:38 project.c2c.local systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Aug 31 20:51:38 project.c2c.local systemd[1]: Unit docker.service entered failed state.

vmadman commented Sep 1, 2015

Aug 31 20:51:38 project.c2c.local systemd[1]: Starting Docker Application Container Engine...
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.516955606-04:00" level=error msg="WARNING: No --storage-opt dm.thinpooldev specified, using loopback; this configuration is strongly discouraged for production use"
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.539035205-04:00" level=error msg="ServeAPI error: No sockets found"
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.617245851-04:00" level=info msg="[graphdriver] using prior storage driver \"devicemapper\""
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.620607898-04:00" level=warning msg="Running modprobe bridge nf_nat br_netfilter failed with message: , error: exit status 1"
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.624593163-04:00" level=info msg="Firewalld running: false"
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.752114083-04:00" level=info msg="Loading containers: start."
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.752424509-04:00" level=info msg="Loading containers: done."
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.752445958-04:00" level=info msg="Daemon has completed initialization"
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.752465223-04:00" level=info msg="Docker daemon" commit="3043001/1.7.1" execdriver=native-0.2 graphdriver=devicemapper version=1.7.1
Aug 31 20:51:38 project.c2c.local systemd[1]: Started Docker Application Container Engine.
Aug 31 20:51:38 project.c2c.local docker[4111]: time="2015-08-31T20:51:38.790898648-04:00" level=fatal msg="Shutting down due to ServeAPI error: No sockets found"
Aug 31 20:51:38 project.c2c.local systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Aug 31 20:51:38 project.c2c.local systemd[1]: Unit docker.service entered failed state.
@kirkhansen

This comment has been minimized.

Show comment
Hide comment
@kirkhansen

kirkhansen Sep 14, 2015

Came across the op's error today. I was wanting to test an automatic docker build test script, and needed to test it when docker was not installed. So I followed the install directions to remove it. Then I tried running the install script again, and the A dependency job for docker.service failed... showed up. I noticed that when I removed docker the first time, I had two yum packages installed docker-engine.x86_64 and docker-selinux.x86_64, but after the re-install the selinux package was not installed. I manually installed that and docker started up. Maybe related, maybe not.

kirkhansen commented Sep 14, 2015

Came across the op's error today. I was wanting to test an automatic docker build test script, and needed to test it when docker was not installed. So I followed the install directions to remove it. Then I tried running the install script again, and the A dependency job for docker.service failed... showed up. I noticed that when I removed docker the first time, I had two yum packages installed docker-engine.x86_64 and docker-selinux.x86_64, but after the re-install the selinux package was not installed. I manually installed that and docker started up. Maybe related, maybe not.

@tungel

This comment has been minimized.

Show comment
Hide comment
@tungel

tungel Sep 16, 2015

Having same problem today:

# systemctl start docker
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.

# systemctl status docker
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: failed (Result: timeout) since Wed 2015-09-16 16:59:47 ACST; 2min 0s ago
     Docs: http://docs.docker.com
 Main PID: 14542

Sep 16 16:58:17 localhost.localdomain systemd[1]: Starting Docker Application Container Engine...
Sep 16 16:58:17 localhost.localdomain docker[14542]: time="2015-09-16T16:58:17.314948677+09:30" level=warning msg="--storage-opt dm.thinpooldev is preferred over --storage-opt dm.datadev or dm.metadatadev"
Sep 16 16:58:17 localhost.localdomain docker[14542]: time="2015-09-16T16:58:17.319520664+09:30" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)"
Sep 16 16:58:17 localhost.localdomain docker[14542]: time="2015-09-16T16:58:17.405736692+09:30" level=info msg="[graphdriver] using prior storage driver \"devicemapper\""
Sep 16 16:58:17 localhost.localdomain docker[14542]: time="2015-09-16T16:58:17.417751496+09:30" level=warning msg="Running modprobe bridge nf_nat br_netfilter failed with message: , error: exit status 1"
Sep 16 16:59:47 localhost.localdomain systemd[1]: docker.service operation timed out. Terminating.
Sep 16 16:59:47 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
Sep 16 16:59:47 localhost.localdomain systemd[1]: Unit docker.service entered failed state.

tungel commented Sep 16, 2015

Having same problem today:

# systemctl start docker
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.

# systemctl status docker
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: failed (Result: timeout) since Wed 2015-09-16 16:59:47 ACST; 2min 0s ago
     Docs: http://docs.docker.com
 Main PID: 14542

Sep 16 16:58:17 localhost.localdomain systemd[1]: Starting Docker Application Container Engine...
Sep 16 16:58:17 localhost.localdomain docker[14542]: time="2015-09-16T16:58:17.314948677+09:30" level=warning msg="--storage-opt dm.thinpooldev is preferred over --storage-opt dm.datadev or dm.metadatadev"
Sep 16 16:58:17 localhost.localdomain docker[14542]: time="2015-09-16T16:58:17.319520664+09:30" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)"
Sep 16 16:58:17 localhost.localdomain docker[14542]: time="2015-09-16T16:58:17.405736692+09:30" level=info msg="[graphdriver] using prior storage driver \"devicemapper\""
Sep 16 16:58:17 localhost.localdomain docker[14542]: time="2015-09-16T16:58:17.417751496+09:30" level=warning msg="Running modprobe bridge nf_nat br_netfilter failed with message: , error: exit status 1"
Sep 16 16:59:47 localhost.localdomain systemd[1]: docker.service operation timed out. Terminating.
Sep 16 16:59:47 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
Sep 16 16:59:47 localhost.localdomain systemd[1]: Unit docker.service entered failed state.
@lsowen

This comment has been minimized.

Show comment
Hide comment
@lsowen

lsowen Oct 8, 2015

I had the same issue. Sorted it by following @kirkhansen fix.

lsowen commented Oct 8, 2015

I had the same issue. Sorted it by following @kirkhansen fix.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 8, 2015

Member

@lsowen @kirkhansen the 1.9 release will have a selinux policy bundled; #15832. For 1.8, manually installing docker-selinux is the solution.

Member

thaJeztah commented Oct 8, 2015

@lsowen @kirkhansen the 1.9 release will have a selinux policy bundled; #15832. For 1.8, manually installing docker-selinux is the solution.

@mlalpho

This comment has been minimized.

Show comment
Hide comment
@mlalpho

mlalpho Jan 31, 2016

I came across this with docker installed on Centos 7 via install instructions.

storage-opt is btrfs

Systemd startup was failing, but docker daemon -D worked fine

Systemd ExecStart=/usr/bin/docker daemon $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY
The command: docker daemon -H fd:// -D -s btrfs -p /var/run/docker.pid
The error: Shutting down due to ServeAPI error: No sockets found

I removed docker-selinux and reinstalled the docker package (which reinstalled docker-selinux).. to no avail.

I ended up removing -H fd:// from my OPTIONS in /etc/sysconfig/docker and the service started as normal.

/etc/sysconfig/docker:

...
OPTIONS=-s btrfs
...
[root@~]# docker info
Containers: 0
Images: 0
Storage Driver: btrfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.20.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 3.452 GiB
Name: ec2.internal
ID: IMN7:BTIJ:T2HH:OKPW:4767:CIKK:XOF2:HBMC:RG3E:W6KT:DD6W:3F2E
[root@~]#

I'm guessing it related to #11135
👍
Cheers!

mlalpho commented Jan 31, 2016

I came across this with docker installed on Centos 7 via install instructions.

storage-opt is btrfs

Systemd startup was failing, but docker daemon -D worked fine

Systemd ExecStart=/usr/bin/docker daemon $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY
The command: docker daemon -H fd:// -D -s btrfs -p /var/run/docker.pid
The error: Shutting down due to ServeAPI error: No sockets found

I removed docker-selinux and reinstalled the docker package (which reinstalled docker-selinux).. to no avail.

I ended up removing -H fd:// from my OPTIONS in /etc/sysconfig/docker and the service started as normal.

/etc/sysconfig/docker:

...
OPTIONS=-s btrfs
...
[root@~]# docker info
Containers: 0
Images: 0
Storage Driver: btrfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.20.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 3.452 GiB
Name: ec2.internal
ID: IMN7:BTIJ:T2HH:OKPW:4767:CIKK:XOF2:HBMC:RG3E:W6KT:DD6W:3F2E
[root@~]#

I'm guessing it related to #11135
👍
Cheers!

@samratBasra

This comment has been minimized.

Show comment
Hide comment
@samratBasra

samratBasra Mar 21, 2016

@jmccormick2001
great solution. just delete the /var/lib/docker directory then reinstall the docker

samratBasra commented Mar 21, 2016

@jmccormick2001
great solution. just delete the /var/lib/docker directory then reinstall the docker

@andreynering

This comment has been minimized.

Show comment
Hide comment
@andreynering

andreynering Jul 6, 2016

I'm having this issue as well. On Fedora Server, after a system upgrade.

I tried to reinstall multiple times, including deleting /var/lib/docker as suggested by some, but with no luck.

andreynering commented Jul 6, 2016

I'm having this issue as well. On Fedora Server, after a system upgrade.

I tried to reinstall multiple times, including deleting /var/lib/docker as suggested by some, but with no luck.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jul 6, 2016

Member

@andreynering could you open a new issue, and provide some more information about your configuration. Also be sure to check the system and daemon logs to see if they contain anything useful

Member

thaJeztah commented Jul 6, 2016

@andreynering could you open a new issue, and provide some more information about your configuration. Also be sure to check the system and daemon logs to see if they contain anything useful

@mojavelinux

This comment has been minimized.

Show comment
Hide comment
@mojavelinux

mojavelinux Aug 27, 2016

I can confirm that the problem (at least on Fedora 21) seems to be the missing docker-io-selinux package (which is called on by the OPTIONS='--selinux-enabled' setting in /etc/sysconfig/docker). Once I installed that package using:

sudo yum install docker-io-selinux

Docker could be started by using sudo systemctl start docker.

mojavelinux commented Aug 27, 2016

I can confirm that the problem (at least on Fedora 21) seems to be the missing docker-io-selinux package (which is called on by the OPTIONS='--selinux-enabled' setting in /etc/sysconfig/docker). Once I installed that package using:

sudo yum install docker-io-selinux

Docker could be started by using sudo systemctl start docker.

@dhan2vijay

This comment has been minimized.

Show comment
Hide comment
@dhan2vijay

dhan2vijay Oct 11, 2017

● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2017-10-11 17:25:49 IST; 5min ago
Docs: http://docs.docker.com
Process: 22899 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE)
Main PID: 22899 (code=exited, status=1/FAILURE)

Oct 11 17:25:49 ose36lab-master.click2cloud.net systemd[1]: Starting Docker Application Container Engine...
Oct 11 17:25:49 ose36lab-master.click2cloud.net dockerd-current[22899]: time="2017-10-11T17:25:49.260877704+05:30" level=fatal msg="Failed to set log opts: unknown log opt 'max-size' for journald log driver"
Oct 11 17:25:49 ose36lab-master.click2cloud.net systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Oct 11 17:25:49 ose36lab-master.click2cloud.net systemd[1]: Failed to start Docker Application Container Engine.
Oct 11 17:25:49 ose36lab-master.click2cloud.net systemd[1]: Unit docker.service entered failed state.
Oct 11 17:25:49 ose36lab-master.click2cloud.net systemd[1]: docker.service failed.

can one help resolving this issue

dhan2vijay commented Oct 11, 2017

● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2017-10-11 17:25:49 IST; 5min ago
Docs: http://docs.docker.com
Process: 22899 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE)
Main PID: 22899 (code=exited, status=1/FAILURE)

Oct 11 17:25:49 ose36lab-master.click2cloud.net systemd[1]: Starting Docker Application Container Engine...
Oct 11 17:25:49 ose36lab-master.click2cloud.net dockerd-current[22899]: time="2017-10-11T17:25:49.260877704+05:30" level=fatal msg="Failed to set log opts: unknown log opt 'max-size' for journald log driver"
Oct 11 17:25:49 ose36lab-master.click2cloud.net systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Oct 11 17:25:49 ose36lab-master.click2cloud.net systemd[1]: Failed to start Docker Application Container Engine.
Oct 11 17:25:49 ose36lab-master.click2cloud.net systemd[1]: Unit docker.service entered failed state.
Oct 11 17:25:49 ose36lab-master.click2cloud.net systemd[1]: docker.service failed.

can one help resolving this issue

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 11, 2017

Member

@dhan2vijay please don't comment on closed tickets to report issues. In your case, the error message shows that it's a problem with the configuration options you're passing to the daemon;

Failed to set log opts: unknown log opt 'max-size' for journald log driver"

The max-size option can only be used for the default json-file logging driver, but is not supported for the journald logging driver https://docs.docker.com/engine/admin/logging/journald/#options

I'm locking the conversation on this issue, to prevent it from collecting non-related comments, but if you suspect you found a bug, feel free to open a new issue and provide the details that are requested in the issue template.

Member

thaJeztah commented Oct 11, 2017

@dhan2vijay please don't comment on closed tickets to report issues. In your case, the error message shows that it's a problem with the configuration options you're passing to the daemon;

Failed to set log opts: unknown log opt 'max-size' for journald log driver"

The max-size option can only be used for the default json-file logging driver, but is not supported for the journald logging driver https://docs.docker.com/engine/admin/logging/journald/#options

I'm locking the conversation on this issue, to prevent it from collecting non-related comments, but if you suspect you found a bug, feel free to open a new issue and provide the details that are requested in the issue template.

@moby moby locked and limited conversation to collaborators Oct 11, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.