Docker Install: Error running DeviceCreate (createPool) dm_task_run failed #6325

Closed
prat0318 opened this Issue Jun 10, 2014 · 66 comments

Projects

None yet
@prat0318

I followed all steps given in the installation guide for 12.04

But, it fails when starting the docker daemon:

~  sudo docker -d
[sudo] password for prat0318:
2014/06/10 02:26:34 docker daemon: 1.0.0 63fe64c; execdriver: native; graphdriver:
[29daee74] +job initserver()
[29daee74.initserver()] Creating server
[29daee74] +job serveapi(unix:///var/run/docker.sock)
2014/06/10 02:26:34 Listening for HTTP on unix (/var/run/docker.sock)
Error running DeviceCreate (createPool) dm_task_run failed
[29daee74] -job initserver() = ERR (1)
2014/06/10 02:26:34 Error running DeviceCreate (createPool) dm_task_run failed

Ubuntu version:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.4 LTS
Release:    12.04
Codename:   precise

Linux Header version:

apt-cache search linux-headers-$(uname -r)
linux-headers-3.8.0-38-generic - Linux kernel headers for version 3.8.0 on 64 bit x86 SMP

Could not find this error anywhere online. Any cues appreciated.

@crosbymichael
Member
@m0ikz
m0ikz commented Jun 12, 2014

I was having the same problem and fixed it adding CONFIG_DM_THIN_PROVISIONING option on the kernel.
https://github.com/dotcloud/docker/tree/master/daemon/graphdriver/devmapper

@prat0318

I changed CONFIG_DM_THIN_PROVISIONING=m to CONFIG_DM_THIN_PROVISIONING=y in /usr/src/linux-headers-3.8.0-29-generic/.config.

It went a bit ahead now, but failed with unknown reason:

$ sudo docker -d
2014/06/13 00:51:17 docker daemon: 1.0.0 63fe64c; execdriver: native; graphdriver: 
[06e7a20c] +job serveapi(unix:///var/run/docker.sock)
[06e7a20c] +job initserver()
[06e7a20c.initserver()] Creating server
2014/06/13 00:51:17 Listening for HTTP on unix (/var/run/docker.sock)
EOF
[06e7a20c] -job initserver() = ERR (1)
2014/06/13 00:51:17 EOF

@m0ikz Thanks for your help. Any idea if you stuck further after that.

@alexlarsson
Contributor

I wonder why it didn't auto-load the dm_thin_pool module.

Can you try running docker -d -D? That should give a bit more info about what goes wrong.

@prat0318

I think even -D did not give anything informative:

014/06/13 20:56:40 docker daemon: 1.0.0 63fe64c; execdriver: native; graphdriver:
[be8a855a] +job serveapi(unix:///var/run/docker.sock)
2014/06/13 20:56:40 Listening for HTTP on unix (/var/run/docker.sock)
[debug] server.go:1114 Registering GET, /events
[debug] server.go:1114 Registering GET, /images/search
[debug] server.go:1114 Registering GET, /images/{name:.*}/get
[debug] server.go:1114 Registering GET, /images/{name:.*}/history
[debug] server.go:1114 Registering GET, /containers/json
[debug] server.go:1114 Registering GET, /containers/{name:.*}/export
[debug] server.go:1114 Registering GET, /containers/{name:.*}/json
[debug] server.go:1114 Registering GET, /info
[debug] server.go:1114 Registering GET, /images/viz
[debug] server.go:1114 Registering GET, /containers/{name:.*}/top
[debug] server.go:1114 Registering GET, /containers/{name:.*}/logs
[debug] server.go:1114 Registering GET, /_ping
[debug] server.go:1114 Registering GET, /images/json
[debug] server.go:1114 Registering GET, /images/{name:.*}/json
[debug] server.go:1114 Registering GET, /containers/{name:.*}/attach/ws
[debug] server.go:1114 Registering GET, /version
[debug] server.go:1114 Registering GET, /containers/ps
[debug] server.go:1114 Registering GET, /containers/{name:.*}/changes
[debug] server.go:1114 Registering POST, /commit
[debug] server.go:1114 Registering POST, /images/{name:.*}/tag
[debug] server.go:1114 Registering POST, /containers/create
[debug] server.go:1114 Registering POST, /containers/{name:.*}/kill
[debug] server.go:1114 Registering POST, /containers/{name:.*}/pause
[debug] server.go:1114 Registering POST, /containers/{name:.*}/stop
[debug] server.go:1114 Registering POST, /containers/{name:.*}/resize
[debug] server.go:1114 Registering POST, /auth
[debug] server.go:1114 Registering POST, /images/create
[debug] server.go:1114 Registering POST, /images/load
[debug] server.go:1114 Registering POST, /images/{name:.*}/push
[debug] server.go:1114 Registering POST, /containers/{name:.*}/start
[debug] server.go:1114 Registering POST, /build
[debug] server.go:1114 Registering POST, /containers/{name:.*}/unpause
[debug] server.go:1114 Registering POST, /containers/{name:.*}/restart
[debug] server.go:1114 Registering POST, /containers/{name:.*}/wait
[be8a855a] +job initserver()
[be8a855a.initserver()] Creating server
[debug] deviceset.go:552 Generated prefix: docker-253:0-274490
[debug] deviceset.go:555 Checking for existence of the pool 'docker-253:0-274490-pool'
EOF
[be8a855a] -job initserver() = ERR (1)
2014/06/13 20:56:40 EOF
@alexlarsson
Contributor

Yeah, that doesn't really give a lot more info. Can you try running docker under strace: strace -o docker.log -f -s 128 docker -d and then attach docker.log (probably compressed)

@obigbando

We bumped into just the same situation as you did. We did the following and the issue seemed to be solved:

  1. check docker.log file, and find out more information about the error
  2. our situation is that our linux kernel and module have in-consist version (3.8 v.s. 3.2)
  3. We install the corresponding version of 3.8 modules at path /lib/modules/3.8.0-31-generic/
@chancez
Contributor
chancez commented Jul 6, 2014

I ran into this when changing my kernel as well. Went from 3.13 on 14.04 to 3.2 trying to enable swap limit support (digital ocean is lagging behind on this). It was fixed by going back to the old kernel however.

@vbatts
Contributor
vbatts commented Jul 10, 2014

Hitting this now on master (aec8d90) and 3.14.9-200.fc20.x86_64

@vbatts
Contributor
vbatts commented Jul 10, 2014

ah, need to include --storage-opt="dm.blocksize=64k" to the daemon flags for the time being

@liutaihua

hello, guys.
I hit this too, on gentoo:

~ # uname -a
Linux 19x004.xd.com 3.10.17-gentoo #1 SMP Thu Nov 7 09:45:08 CST 2013 x86_64 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz GenuineIntel GNU/Linux

~ # docker -v
Docker version 1.1.0, build 79812e3

~ # docker -D -d
2014/07/11 12:02:45 docker daemon: 1.1.0 79812e3; execdriver: native; graphdriver:
[a0f55212] +job serveapi(unix:///var/run/docker.sock)
[a0f55212] +job initserver()
[a0f55212.initserver()] Creating server
[debug] deviceset.go:552 Generated prefix: docker-8:3-176964760
[debug] deviceset.go:555 Checking for existence of the pool 'docker-8:3-176964760-pool'
[debug] deviceset.go:574 Pool doesn't exist. Creating it.
[debug] deviceset.go:448 libdevmapper(3): ioctl/libdm-iface.c:1705 (-1) device-mapper: reload ioctl on docker-8:3-176964760-pool failed: Invalid argument
Error running DeviceCreate (createPool) dm_task_run failed
[a0f55212] -job initserver() = ERR (1)
2014/07/11 12:02:45 Error running DeviceCreate (createPool) dm_task_run failed

@unclejack
Contributor

For those who are running into this problem, could you try running this [1] script and comment if you're seeing anything missing in the devicemapper section, please?

[1] https://raw.githubusercontent.com/dotcloud/docker/master/contrib/check-config.sh

@alexlarsson alexlarsson was unassigned by prat0318 Jul 24, 2014
@marcellodesales

I broke a server after trying to install Shipyard... How bad is this? My kernel's version is way older!!! Is there any other workaround to other than reinstall the host with the latest REHL7? @unclejack, what do you think?

[root@pppdc9prd4c4 ~]# . check-config.sh
warning: /proc/config.gz does not exist, searching other paths for kernel config...
info: reading kernel config from /boot/config-2.6.32-431.1.2.el6.x86_64 ...

Generally Necessary:

  • cgroup hierarchy: properly mounted [/cgroup]
  • CONFIG_NAMESPACES: enabled
  • CONFIG_NET_NS: enabled
  • CONFIG_PID_NS: enabled
  • CONFIG_IPC_NS: enabled
  • CONFIG_UTS_NS: enabled
  • CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
  • CONFIG_CGROUPS: enabled
  • CONFIG_CGROUP_CPUACCT: enabled
  • CONFIG_CGROUP_DEVICE: enabled
  • CONFIG_CGROUP_FREEZER: enabled
  • CONFIG_CGROUP_SCHED: enabled
  • CONFIG_MACVLAN: enabled
  • CONFIG_VETH: enabled
  • CONFIG_BRIDGE: enabled
  • CONFIG_NF_NAT_IPV4: missing
  • CONFIG_IP_NF_TARGET_MASQUERADE: enabled
  • CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: missing
  • CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
  • CONFIG_NF_NAT: enabled
  • CONFIG_NF_NAT_NEEDED: enabled

Optional Features:

  • CONFIG_MEMCG_SWAP: missing
  • CONFIG_RESOURCE_COUNTERS: enabled
  • Storage Drivers:
    • "aufs":
      • CONFIG_AUFS_FS: missing
    • "btrfs":
      • CONFIG_BTRFS_FS: enabled
    • "devicemapper":
      • CONFIG_BLK_DEV_DM: enabled
      • CONFIG_DM_THIN_PROVISIONING: enabled
      • CONFIG_EXT4_FS: enabled
@vbatts
Contributor
vbatts commented Aug 6, 2014

@marcellodesales did you add --storage-opt="dm.blocksize=64k" to /etc/sysconfig/docker (if you want to preserve your /var/lib/docker)? The RHEL6 kernel is not the issue here.

@marcellodesales

@vbatts, I tried different options here and I couldn't get it running :( Could you share an example of how the file looks like with that... The following are my attempts to get it working...

[root@pppdc9prd4cf mdesales]# cat /etc/sysconfig/docker
# /etc/sysconfig/docker
#
# Other arguments to pass to the docker daemon process
# These will be parsed by the sysv initscript and appended
# to the arguments list passed to docker -d
export HTTP_PROXY=http://ie.xxxxxx.net:80
other_args="--storage-opt=dm.blocksize=64k"

I tried to execute it directly in the command-line as well...

[root@pppdc9prd4cf mdesales]# docker --storage-opt=dm.blocksize=64k -d
2014/08/07 13:45:28 WARNING: You are running linux kernel version 2.6.32-431.1.2.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.8.0.
2014/08/07 13:45:28 docker daemon: 1.0.0 63fe64c/1.0.0; execdriver: native; graphdriver:
[8925db36] +job initserver()
[8925db36.initserver()] Creating server
[8925db36] +job serveapi(unix:///var/run/docker.sock)
2014/08/07 13:45:28 Listening for HTTP on unix (/var/run/docker.sock)
Unknown option dm.blocksize
[8925db36] -job initserver() = ERR (1)

Then, I tried with the contents updated...

2014/08/07 13:47:15 [root@pppdc9prd4cf mdesales]# docker -d
2014/08/07 13:47:24 WARNING: You are running linux kernel version 2.6.32-431.1.2.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.8.0.
2014/08/07 13:47:24 docker daemon: 1.0.0 63fe64c/1.0.0; execdriver: native; graphdriver:
[9d2774fc] +job initserver()
[9d2774fc.initserver()] Creating server
[9d2774fc] +job serveapi(unix:///var/run/docker.sock)
2014/08/07 13:47:24 Listening for HTTP on unix (/var/run/docker.sock)
Error running DeviceCreate (createPool) dm_task_run failed
[9d2774fc] -job initserver() = ERR (1)
2014/08/07 13:47:24 Error running DeviceCreate (createPool) dm_task_run failed
[root@pppdc9prd4cf mdesales]# service docker start
Starting docker:                                       [  OK  ]

With those attempts, the daemon does not start, even though it is reported to be running...

[root@pppdc9prd4cf mdesales]# ps aux | grep docker
root      3112  0.0  0.0 103252   820 pts/0    S+   13:47   0:00 grep docker

I tried other formats :S, removing the quotes, etc... And defining it in a new line, but docker complained about the "format and command"...

[root@pppdc9prd4cf mdesales]# cat /etc/sysconfig/docker
# /etc/sysconfig/docker
#
# Other arguments to pass to the docker daemon process
# These will be parsed by the sysv initscript and appended
# to the arguments list passed to docker -d
export HTTP_PROXY=http://qypprdproxy02.ie.intuit.net:80
other_args=--storage-opt="dm.blocksize=64k"
[root@pppdc9prd4cf mdesales]#

Failed with command not found...

[root@pppdc9prd4cf mdesales]# cat /etc/sysconfig/docker
# /etc/sysconfig/docker
#
# Other arguments to pass to the docker daemon process
# These will be parsed by the sysv initscript and appended
# to the arguments list passed to docker -d
export HTTP_PROXY=http://qypprdproxy02.ie.intuit.net:80
--storage-opt="dm.blocksize=64k"
other_args=""
[root@pppdc9prd4cf mdesales]#
@marcellodesales

Also, after the attempts to run this, I can't start docker on my machine anymore :( I reverted the config and tried executing it again and Docker can't start...

[root@pppdc9prd4cf mdesales]# cat /etc/sysconfig/docker
# /etc/sysconfig/docker
#
# Other arguments to pass to the docker daemon process
# These will be parsed by the sysv initscript and appended
# to the arguments list passed to docker -d
export HTTP_PROXY=http://qypprdproxy02.ie.intuit.net:80
other_args=""

Trying to use docker -d

[root@pppdc9prd4cf mdesales]# sudo docker -d
2014/08/07 14:05:12 WARNING: You are running linux kernel version 2.6.32-431.1.2.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.8.0.
2014/08/07 14:05:12 docker daemon: 1.0.0 63fe64c/1.0.0; execdriver: native; graphdriver:
[6faa772f] +job serveapi(unix:///var/run/docker.sock)
[6faa772f] +job initserver()
[6faa772f.initserver()] Creating server
2014/08/07 14:05:12 Listening for HTTP on unix (/var/run/docker.sock)
Error running DeviceCreate (createPool) dm_task_run failed
[6faa772f] -job initserver() = ERR (1)
2014/08/07 14:05:12 Error running DeviceCreate (createPool) dm_task_run failed
@phemmer
Contributor
phemmer commented Aug 18, 2014

I received this error when docker used up all it's disk space. I went and removed a bunch of images to free some space up, but any time I tried to start a container, it threw the Error running DeviceCreate (createPool) dm_task_run failed error.
I tried stopping docker and starting it back up, and now it won't even start.

Going to have to wipe the docker partition and start over.

@alexlarsson
Contributor

#4202 has backend-specific operations, including pool resizing. That would partly help the disk space issue (by allowing you to avoid it ahead of time).

I think what is happening here is that the pool gets into some kind of error state that we have to handle, maybe with some kind of pool-fsck command.

@marcellodesales

@phemmer, I can confirm the error is related to the partition being out of space!

@fosstamil

I have the same problem.

Error running DeviceCreate (createPool) dm_task_run failed

How do I recover Docker containers and images from the disk?

@marcellodesales

@fosstamil Did you check if you have enough space in the partition where you images are created? The images are stored under /var/lib/docker ? (or any other dir you configured the libs to be?)

@fracklen

i broke a server (digital-ocean), moving it to another data-center.

@unclejack output from check-config:


Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_MACVLAN: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled

Optional Features:
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_RESOURCE_COUNTERS: enabled
- CONFIG_CGROUP_PERF: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
    - CONFIG_EXT4_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled

Tried all the things above

  • Enough disk space
  • Tried deleting /var/lib/docker
  • uninstall/reinstall docker

$ docker -d -D
2014/09/29 17:36:47 docker daemon: 1.2.0 fa7b24f; execdriver: native; graphdriver:
[040c0d99] +job serveapi(unix:///var/run/docker.sock)
[info] Listening for HTTP on unix (/var/run/docker.sock)
[debug] server.go:1151 Registering GET, /images/search
[debug] server.go:1151 Registering GET, /containers/ps
[debug] server.go:1151 Registering GET, /containers/{name:.}/export
[debug] server.go:1151 Registering GET, /images/viz
[debug] server.go:1151 Registering GET, /info
[debug] server.go:1151 Registering GET, /version
[debug] server.go:1151 Registering GET, /images/json
[debug] server.go:1151 Registering GET, /images/{name:.
}/get
[debug] server.go:1151 Registering GET, /images/{name:.}/json
[debug] server.go:1151 Registering GET, /containers/json
[debug] server.go:1151 Registering GET, /containers/{name:.
}/changes
[debug] server.go:1151 Registering GET, /events
[debug] server.go:1151 Registering GET, /containers/{name:.}/top
[debug] server.go:1151 Registering GET, /containers/{name:.
}/json
[debug] server.go:1151 Registering GET, /images/{name:.}/history
[debug] server.go:1151 Registering GET, /_ping
[debug] server.go:1151 Registering GET, /containers/{name:.
}/attach/ws
[debug] server.go:1151 Registering GET, /containers/{name:.}/logs
[debug] server.go:1151 Registering POST, /commit
[debug] server.go:1151 Registering POST, /images/load
[debug] server.go:1151 Registering POST, /containers/{name:.
}/restart
[debug] server.go:1151 Registering POST, /containers/{name:.}/attach
[debug] server.go:1151 Registering POST, /containers/{name:.
}/stop
[debug] server.go:1151 Registering POST, /containers/{name:.}/wait
[debug] server.go:1151 Registering POST, /auth
[debug] server.go:1151 Registering POST, /images/{name:.
}/tag
[debug] server.go:1151 Registering POST, /containers/{name:.}/unpause
[debug] server.go:1151 Registering POST, /containers/{name:.
}/start
[debug] server.go:1151 Registering POST, /images/create
[debug] server.go:1151 Registering POST, /images/{name:.}/push
[debug] server.go:1151 Registering POST, /containers/{name:.
}/resize
[debug] server.go:1151 Registering POST, /containers/{name:.}/copy
[debug] server.go:1151 Registering POST, /build
[debug] server.go:1151 Registering POST, /containers/create
[debug] server.go:1151 Registering POST, /containers/{name:.
}/kill
[debug] server.go:1151 Registering POST, /containers/{name:.}/pause
[debug] server.go:1151 Registering DELETE, /containers/{name:.
}
[debug] server.go:1151 Registering DELETE, /images/{name:.*}
[debug] server.go:1151 Registering OPTIONS,
[debug] server.go:1241 docker group found. gid: 999
[debug] deviceset.go:555 Generated prefix: docker-253:1-1966719
[debug] deviceset.go:558 Checking for existence of the pool 'docker-253:1-1966719-pool'
[debug] deviceset.go:577 Pool doesn't exist. Creating it.
[debug] deviceset.go:451 libdevmapper(3): ioctl/libdm-iface.c:1768 (-1) device-mapper: reload ioctl on docker-253:1-1966719-pool failed: Invalid argument
2014/09/29 17:36:47 Error running DeviceCreate (createPool) dm_task_run failed


Help?
@unclejack
Contributor

@fracklen Please upgrade your kernel to something supported. The LTS trusty kernel image is also available for 12.04. 3.8 is no longer supported by anyone, including Canonical.

@fracklen

@unclejack upgraded to 3.13 - no luck - what's next?

info: reading kernel config from /boot/config-3.13.0-36-generic ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_MACVLAN: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled

Optional Features:
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_RESOURCE_COUNTERS: enabled
- CONFIG_CGROUP_PERF: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
    - CONFIG_EXT4_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled
@xiaokangwang

Are you using digital ocean?

I was once suffering from the same problem until I run

locate ".config"

and then switched my kernel to one shown at the last command.

err4

err5

This worked for me, please try if this works for you as well.

@fracklen
fracklen commented Oct 7, 2014

@xiaokangwang yep. It appears to be a problem with DigitalOcean.

Somehow there was an inconsistency between the actual kernel and the one Ubuntu had installed / used by grub.

We switched the kernel in their interface and the problem persisted. Only when we moved the image to another vps and in the process updated the kernel - both grub and DO-interface, the problem went away.

@saidimu
saidimu commented Oct 8, 2014

I had the same "mixed up kernels" issue on DigitalOcean after migrating an
image to a new host.

I fixed it by installing the kernel specified by uname -r
On Oct 7, 2014 11:02 AM, "Martin Neiiendam" notifications@github.com
wrote:

@xiaokangwang https://github.com/xiaokangwang yep. It appears to be a
problem with DigitalOcean.

Somehow there was an inconsistency between the actual kernel and the one
Ubuntu had installed / used by grub.

We switched the kernel in their interface and the problem persisted. Only
when we moved the image to another vps and in the process updated the
kernel - both grub and DO-interface, the problem went away.


Reply to this email directly or view it on GitHub
#6325 (comment).

@vbatts
Contributor
vbatts commented Oct 14, 2014

It is my feeling that the ops issue was that createPool failed when default pool blocksize had been altered. This default has since been rolled back (d715803).

I'm in favor of closing this issue. /cc @unclejack

@peterlitvak

I have the same issue
Ubuntu 14.04
kernel 3.13.0-32-generic
The ubuntu is running in VirtualBox is that matters and there is enough space on the partition (since some people attributed the problem to the space availability issue)
Nothing from above worked.
Any more suggestions?

@vbatts
Contributor
vbatts commented Nov 18, 2014

@peterlitvak could you provide a selection of the daemon logs with debugging on that shows the error; the output of docker version and docker info please?

@peterlitvak

The daemon docker -d just fails to start so no logs or docker info output are available. The error is the same as in the originator's post.
I think the problem is in the fact that dm_thin_pool module is not loaded since dmsetup targets lists only the following:
sriped
linear
error
I don't know much about DM unfortunately to troubleshoot this.

@peterlitvak

Just installed Ubuntu 14.10 in a new VM and docker seems to install and run (at least docker info now works) just fine. The dmsetup targets still show only those three so I guess this is not an indicator of a problem.

@tnachen
tnachen commented Dec 5, 2014

Running into the same problem with Ubuntu 14.10 and VirtualBox as well, not sure what to do.

@unclejack
Contributor

devicemapper issues review session with @vbatts

This issue has been reported a long time ago. If you happen to be affected by this, please provide the following bits of information which are mandatory for looking into this issue:

  • uname -a, docker version and docker info output
  • step by step instructions on how to reproduce the problem

In the meantime, I'm going to close this issue.

@unclejack unclejack closed this Jan 7, 2015
@Lawouach

Hi there,

Installed a fresh Ubuntu server 14.04.1 on a server with two disks partitionned with a two LVM logical volumes (one for my system and one for the swap).

I installed lxc-docker 1.4.1 and when I try to start the daemon, here is what I got:

$ sudo docker  -d
INFO[0000] +job serveapi(unix:///var/run/docker.sock)   
FATA[0000] Error running DeviceCreate (CreatePool) dm_task_run failed 
$ uname -a
Linux int-docker0 3.16.0-29-generic #39-Ubuntu SMP Mon Dec 15 22:27:29 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
$ docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
FATA[0000] Cannot connect to the Docker daemon. Is 'docker -d' running on this host?
$ docker info
FATA[0000] Cannot connect to the Docker daemon. Is 'docker -d' running on this host?
[   21.414439] device-mapper: table: 252:2: thin-pool: unknown target type
[   21.414444] device-mapper: ioctl: error adding target to table
[   21.417047] init: docker main process (1280) terminated with status 1
[   21.417057] init: docker main process ended, respawning

I have no doubt it could be my system which is somewhat broken. Still, wanted to update this issue.

@kk86bioinfo

Hi,

I am following standard steps for Docker installation in https://docs.docker.com/installation/ubuntulinux/#ubuntu-precise-1204-lts-64-bit

I am too experience with such issue in Ubuntu 12.04 LTS.
docker -d
INFO[0000] +job serveapi(unix:///var/run/docker.sock)   
FATA[0000] Error running DeviceCreate (CreatePool) dm_task_run failed 
uname -a
Linux systeomics-cloud.org 3.8.0-38-generic #56~precise1-Ubuntu SMP Thu Mar 13 16:22:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
sudo docker info
FATA[0000] Cannot connect to the Docker daemon. Is 'docker -d' running on this host? 
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
FATA[0000] Cannot connect to the Docker daemon. Is 'docker -d' running on this host? 
@Lawouach

A follow up from my side. I reinstalled Ubuntu 14.04.1 from scratch but this time, I deleted all my LVM logical volumes and groups. I simply used one of my two disks. Now docker works fine.

I'm not actually sure it was an issue with docker mind you since, after the first installation using LVM volumes, I had this error:

http://askubuntu.com/questions/468466/why-this-occurs-error-diskfilter-writes-are-not-supported

I am now wondering if either one of my disks isn't at fault, or simply LVM's support in Ubuntu isn't that reliable.

@attilagyorffy

I m having the same error:

~> sudo strace -o docker.log -f -s 128 docker -d -D
INFO[0000] +job serveapi(unix:///var/run/docker.sock)   
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1750 (4) dm version   OF   [16384] (*1) 
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1750 (4) dm version   OF   [16384] (*1) 
WARN[0000] WARNING: Udev sync is not supported. This will lead to unexpected behavior, data loss and errors 
DEBU[0000] devicemapper: udev sync support: false       
DEBU[0000] Generated prefix: docker-253:1-132227        
DEBU[0000] Checking for existence of the pool 'docker-253:1-132227-pool' 
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1750 (4) dm info docker-253:1-132227-pool  OF   [16384] (*1) 
DEBU[0000] Pool doesn't exist. Creating it.             
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1750 (4) dm create docker-253:1-132227-pool  OF   [16384] (*1) 
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1750 (4) dm reload docker-253:1-132227-pool  OF   [16384] (*1) 
DEBU[0000] libdevmapper(3): ioctl/libdm-iface.c:1768 (-1) device-mapper: reload ioctl on docker-253:1-132227-pool failed: Invalid argument 
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1359 (0) <backtrace> 
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1750 (4) dm remove docker-253:1-132227-pool  OF   [16384] (*1) 
DEBU[0000] libdevmapper(6): libdm-common.c:1351 (4) docker-253:1-132227-pool: Stacking NODE_DEL [verify_udev] 
DEBU[0000] libdevmapper(6): libdm-common.c:1351 (4) docker-253:1-132227-pool: Processing NODE_DEL [verify_udev] 
FATA[0000] Error running DeviceCreate (CreatePool) dm_task_run failed 
attila@nl ~> docker --version
Docker version 1.5.0, build a8a31ef
attila@nl ~> uname -r
3.16.0-29-generic
attila@nl ~> docker info
FATA[0000] Cannot connect to the Docker daemon. Is 'docker -d' running on this host?

/CC @unclejack here's the debug information you mentioned.

Provided the issue was closed and there's a solution to the problem, I'd love to know. Thanks.

@marvell
marvell commented Feb 25, 2015

Same problem. DigitalOcean, ubuntu 14.04.2, kernel 3.13.0-45-memlimit, docker 1.5.0.

@marvell
marvell commented Feb 25, 2015

I changed the kernel on default in DO panel and it helped to me.

@attilagyorffy

@marvell can you confirm the version of the kernel you're using? Thanks.

@marvell
marvell commented Feb 25, 2015

@liquid Your droplet's current kernel: Ubuntu 14.04 x64 vmlinuz-3.13.0-43-generic

@attilagyorffy

@marvell I've changed back to the original kernel and I've still got no luck :(

@marvell
marvell commented Feb 25, 2015

@liquid Try reinstall docker:

sudo apt-get purge lxc-docker
sudo apt-get install -y lxc-docker
@hordemark

uname -a
Linux DLN-VM-CV14 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

docker --version
Docker version 1.4.1, build 5bc2ff8/1.4.1

@francisdb

this helped for me:
(after switching to the latest kernel on DigitalOcean)

sudo apt-get install linux-image-extra-`uname -r`
@sahus5
sahus5 commented Apr 30, 2015

@vbatts
I am hitting this issue on docker version 1.4.1
Error log:
015-04-30T02:12:13.357547+00:00 ogden-newyork docker[4597]: Error running DeviceCreate (createSnapDevice) dm_task_run failed
2015-04-30T02:12:13.357866+00:00 ogden-newyork docker[4597]: time="2015-04-30T02:12:13Z" level="error" msg="Handler for POST /containers/create returned error: Error running DeviceCreate (createSnapDevice) dm_task_run failed"
2015-04-30T02:12:13.358016+00:00 ogden-newyork docker[4597]: time="2015-04-30T02:12:13Z" level="error" msg="HTTP Error: statusCode=500 Error running DeviceCreate (createSnapDevice) dm_task_run failed"
2015-04-30T02:12:13.358228+00:00 ogden-newyork kernel: [ 160.394915] device-mapper: thin: Creation of new snapshot 15 of device 12 failed.

Can you please tell me a way to resolve this and to start my container?

docker info
Containers: 1
Images: 7
Storage Driver: devicemapper
Pool Name: docker-254:0-687865984-pool
Pool Blocksize: 65.54 kB
Data file: /var/lib/docker/devicemapper/devicemapper/data
Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 0 B
Data Space Total: 107.4 GB
Metadata Space Used: 577.5 kB
Metadata Space Total: 2.147 GB
Library Version: 1.03.01 (2011-10-15)
Execution Driver: native-0.2
Kernel Version: 3.12.39-47-default
Operating System: SUSE Linux Enterprise Server 12
CPUs: 8
Total Memory: 58.7 GiB
Name: ogden-newyork
ID: HX6C:M3IK:E5HF:BBOQ:GWIH:ASGW:GAHJ:LZAZ:KTMM:Z66Z:KP5J:4BC5
WARNING: No swap limit support
ogden-newyork:/var/lib/docker/devicemapper/devicemapper #

@vbatts
Contributor
vbatts commented Apr 30, 2015

@sahus5 is your root file system on lvm? If so, it would be best to make an actual volume block device that is a thin pool to start docker with.

@jaredly
jaredly commented May 14, 2015

I ran into this error, and the problem was that my root fs had run out of space

@sahus5
sahus5 commented May 14, 2015

One of the application was writing logs to rootfs and it ran out of space.
Changed the application to not to write to rootfs resolved the issue. Might be making use of readonly rootfs will be helpful from docker version 1.5 on.

@fenstrat

Can also confirm that like @francisdb with the latest kernel the fix was:

sudo apt-get install linux-image-extra-`uname -r`
@hnasrullakhan

Hi,
I see the error,

How did you guys fix it.

2015/07/10 14:06:48 docker daemon: 1.3.3 17fec36-dirty; execdriver: native; graphdriver:
[6ac62fa0] +job serveapi(unix:///var/run/docker.sock)
[debug] server.go:1300 Registering GET, /images/viz
[debug] server.go:1300 Registering GET, /containers/{name:.}/changes
[debug] server.go:1300 Registering GET, /containers/ps
[debug] server.go:1300 Registering GET, /containers/{name:.
}/attach/ws
[debug] server.go:1300 Registering GET, /ping
[debug] server.go:1300 Registering GET, /events
[debug] server.go:1300 Registering GET, /images/get
[debug] server.go:1300 Registering GET, /images/{name:.
}/json
[debug] server.go:1300 Registering GET, /images/{name:.}/get
[debug] server.go:1300 Registering GET, /containers/{name:.
}/json
[debug] server.go:1300 Registering GET, /containers/{name:.}/logs
[debug] server.go:1300 Registering GET, /info
[debug] server.go:1300 Registering GET, /version
[debug] server.go:1300 Registering GET, /images/json
[debug] server.go:1300 Registering GET, /images/search
[debug] server.go:1300 Registering GET, /images/{name:.
}/history
[debug] server.go:1300 Registering GET, /containers/json
[debug] server.go:1300 Registering GET, /containers/{name:.}/export
[debug] server.go:1300 Registering GET, /containers/{name:.
}/top
[debug] server.go:1300 Registering POST, /exec/{name:.}/resize
[debug] server.go:1300 Registering POST, /containers/{name:.
}/kill
[debug] server.go:1300 Registering POST, /containers/{name:.}/restart
[debug] server.go:1300 Registering POST, /containers/{name:.
}/wait
[debug] server.go:1300 Registering POST, /containers/create
[debug] server.go:1300 Registering POST, /containers/{name:.}/pause
[debug] server.go:1300 Registering POST, /containers/{name:.
}/start
[debug] server.go:1300 Registering POST, /auth
[debug] server.go:1300 Registering POST, /build
[debug] server.go:1300 Registering POST, /images/{name:.}/push
[debug] server.go:1300 Registering POST, /containers/{name:.
}/stop
[debug] server.go:1300 Registering POST, /containers/{name:.}/resize
[debug] server.go:1300 Registering POST, /containers/{name:.
}/attach
[debug] server.go:1300 Registering POST, /containers/{name:.}/copy
[debug] server.go:1300 Registering POST, /containers/{name:.
}/exec
[debug] server.go:1300 Registering POST, /images/create
[debug] server.go:1300 Registering POST, /images/load
[debug] server.go:1300 Registering POST, /images/{name:.}/tag
[debug] server.go:1300 Registering POST, /exec/{name:.
}/start
[debug] server.go:1300 Registering POST, /commit
[debug] server.go:1300 Registering POST, /containers/{name:.}/unpause
[debug] server.go:1300 Registering DELETE, /containers/{name:.
}
[debug] server.go:1300 Registering DELETE, /images/{name:._}
[debug] server.go:1300 Registering OPTIONS,
[debug] server.go:1471 Warning: could not chgrp /var/run/docker.sock to docker: Group docker not found
[debug] deviceset.go:565 Generated prefix: docker-0:34-55785
[debug] deviceset.go:568 Checking for existence of the pool 'docker-0:34-55785-pool'
[debug] deviceset.go:587 Pool doesn't exist. Creating it.
[debug] deviceset.go:455 libdevmapper(3): ioctl/libdm-iface.c:1705 (-1) device-mapper: reload ioctl on docker-0:34-55785-pool failed: Invalid argument
2015/07/10 14:06:48 Error running DeviceCreate (createPool) dm_task_run failed

[Switch_1_RP_0:/usr/bin]$ uname -a
Linux Switch_1_RP_0 3.10.65.cge-rt68 #4 SMP Thu Jul 9 11:55:57 EDT 2015 x86_64 GNU/Linux

[Switch_1_RP_0:/usr/bin]$ docker --version
Docker version 1.3.3, build 17fec36-dirty
[Switch_1_RP_0:/usr/bin]$

[Switch_1_RP_0:/usr/bin]$ docker info
2015/07/10 14:10:55 Cannot connect to the Docker daemon. Is 'docker -d' running on this host?
[Switch_1_RP_0:/usr/bin]$

Please Help, Stuck here from long

Nasrulla

@wolfch
wolfch commented Jul 29, 2015

I am having this issue on CentOS-7 - can anyone help?

uname -a: Linux localhost.localdomain 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

docker version:

Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): ba1f6c3/1.6.2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): ba1f6c3/1.6.2
OS/Arch (server): linux/amd64

docker info:

Containers: 0
Images: 1
Storage Driver: devicemapper
Pool Name: docker-253:0-17401461-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 0 B
Data Space Total: 107.4 GB
Data Space Available: 3.163 GB
Metadata Space Used: 577.5 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.7.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 1
Total Memory: 3.703 GiB
Name: localhost.localdomain
ID: EE5E:QXCN:PGH3:O4FE:FWVW:HVQF:2EJ2:QXUZ:B6BK:GYPD:VN7X:BXIE
Debug mode (server): true
Debug mode (client): false
Fds: 12
Goroutines: 19
System Time: Wed Jul 29 11:19:28 EDT 2015
EventsListeners: 0
Init SHA1: 37963b57dd7d1572d07669a260ee8d68d1baddc6
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker

Steps to reproduce:
1.) start docker daemon in foreground with debug output

docker -d -D

2.) run "docker pull hello-world" in another terminal window

These steps were all run as root and root is a member of the "docker" group.

Is there any hope of running docker on CentOS-7?

@max-m
max-m commented Aug 2, 2015

this helped for me:
(after switching to the latest kernel on DigitalOcean)

sudo apt-get install linux-image-extra-uname -r

@francisdb thank you! :)

@beetree
beetree commented Aug 20, 2015

Also having the issue on CentOS 7.

docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

Daemon start:

docker --storage-opt dm.basesize=2G -d
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
INFO[0000] [graphdriver] using prior storage driver "devicemapper" 
WARN[0000] Running modprobe bridge nf_nat failed with message: , error: exit status 1 
INFO[0000] Loading containers: start.                   
...
INFO[0000] Loading containers: done.                    
INFO[0000] Daemon has completed initialization          
INFO[0000] Docker daemon                                 commit=786b29d execdriver=native-0.2 graphdriver=devicemapper version=1.7.1

Error generated by:

docker run -it centos
Unable to find image 'centos:latest' locally
latest: Pulling from centos
f1b10cd84249: Download complete 
f1b10cd84249: Error downloading dependent layers 
c852f6d61e65: Downloading [===>                                               ] 4.295 MB/62.89 MB
7322fbe74aa5: Error pulling image (latest) from centos, Driver devicemapper failed to create image rootfs f1b10cd842498c23d206ee0cbeaa9de8d2ae09ff3c7af2723a9e337a6965d639: Error running DeviceCreate (createSnapDError pulling image (latest) from centos, Driver devicemapper failed to create image rootfs f1b10cd842498c23d206ee0cbeaa9de8d2ae09ff3c7af2723a9e337a6965d639: Error running DeviceCreate (createSnapDevice) dm_task_run failed

Output inside daemon:

INFO[0031] POST /v1.19/containers/create                
ERRO[0031] Handler for POST /containers/create returned error: No such image: centos (tag: latest) 
ERRO[0031] HTTP Error                                    err=No such image: centos (tag: latest) statusCode=404
INFO[0031] POST /v1.19/images/create?fromImage=centos&tag=latest 
ERRO[0033] Error from V2 registry: Driver devicemapper failed to create image rootfs f1b10cd842498c23d206ee0cbeaa9de8d2ae09ff3c7af2723a9e337a6965d639: Error running DeviceCreate (createSnapDevice) dm_task_run failed 
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x62d76a]

goroutine 66 [running]:
bufio.(*Writer).flush(0xc2084550c0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:530 +0xda
bufio.(*Writer).Flush(0xc2084550c0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:519 +0x3a
net/http.(*response).Flush(0xc208124a00)
    /usr/local/go/src/net/http/server.go:1047 +0x4c
github.com/docker/docker/pkg/ioutils.(*WriteFlusher).Write(0xc208465560, 0xc20877d080, 0xbc, 0x14c, 0xbc, 0x0, 0x0)
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/ioutils/writeflusher.go:21 +0x145
github.com/docker/docker/pkg/progressreader.(*Config).Read(0xc2087d2690, 0xc2087d8000, 0x8000, 0x8000, 0x4000, 0x0, 0x0)
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/progressreader/progressreader.go:37 +0x2e5
io.Copy(0x7f6ceef8f2a8, 0xc208036c60, 0x7f6ceefa1b88, 0xc2087d2690, 0x49898a, 0x0, 0x0)
    /usr/local/go/src/io/io.go:362 +0x1f6
github.com/docker/docker/graph.func·008(0xc2083dd4f0, 0x0, 0x0)
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/graph/pull.go:602 +0xd42
github.com/docker/docker/graph.func·009(0xc2083dd4f0)
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/graph/pull.go:626 +0x2f
created by github.com/docker/docker/graph.(*TagStore).pullV2Tag
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/graph/pull.go:627 +0x2671

goroutine 1 [chan receive]:
main.mainDaemon()
    /root/rpmbuild/BUILD/docker-engine/docker/daemon.go:179 +0x10a8
main.main()
    /root/rpmbuild/BUILD/docker-engine/docker/docker.go:93 +0x6b4

goroutine 5 [syscall]:
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 11 [IO wait]:
net.(*pollDesc).Wait(0xc20814e060, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20814e060, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc20814e000, 0x0, 0x7f6ceef8f1a0, 0xc2084525c0)
    /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*UnixListener).AcceptUnix(0xc2081ae0a0, 0xc20844d520, 0x0, 0x0)
    /usr/local/go/src/net/unixsock_posix.go:282 +0x56
net.(*UnixListener).Accept(0xc2081ae0a0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/unixsock_posix.go:293 +0x4c
github.com/docker/docker/pkg/listenbuffer.(*defaultListener).Accept(0xc2081ae0c0, 0x0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/listenbuffer/buffer.go:71 +0x67
net/http.(*Server).Serve(0xc20819c060, 0x7f6ceef96d70, 0xc2081ae0c0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1728 +0x92
github.com/docker/docker/api/server.(*HttpServer).Serve(0xc2081ae240, 0x0, 0x0)
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/server.go:128 +0x4d
github.com/docker/docker/api/server.func·002(0x7f6ceef96dd0, 0xc2081ae240)
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/server.go:104 +0x40
created by github.com/docker/docker/api/server.(*Server).ServeApi
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/server.go:108 +0x6ac

goroutine 9 [chan receive]:
github.com/docker/docker/api/server.(*Server).ServeApi(0xc2080f4480, 0xc2080f6c40, 0x1, 0x1, 0x0, 0x0)
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/server.go:113 +0x74f
main.func·003()
    /root/rpmbuild/BUILD/docker-engine/docker/daemon.go:129 +0x5f
created by main.mainDaemon
    /root/rpmbuild/BUILD/docker-engine/docker/daemon.go:135 +0x6b6

goroutine 10 [chan receive]:
github.com/docker/docker/daemon.func·021()
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/debugtrap.go:17 +0x5c
created by github.com/docker/docker/daemon.setupSigusr1Trap
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/debugtrap.go:20 +0x18e

goroutine 36 [chan receive]:
database/sql.(*DB).connectionOpener(0xc208183a40)
    /usr/local/go/src/database/sql/sql.go:589 +0x4c
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:452 +0x31c

goroutine 37 [IO wait]:
net.(*pollDesc).Wait(0xc2081441b0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2081441b0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readMsg(0xc208144150, 0xc20843dc00, 0x10, 0x10, 0xc208444420, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ...)
    /usr/local/go/src/net/fd_unix.go:296 +0x54e
net.(*UnixConn).ReadMsgUnix(0xc208037028, 0xc20843dc00, 0x10, 0x10, 0xc208444420, 0x1000, 0x1000, 0x5c, 0xc20843d9c4, 0x4, ...)
    /usr/local/go/src/net/unixsock_posix.go:147 +0x167
github.com/godbus/dbus.(*oobReader).Read(0xc208444400, 0xc20843dc00, 0x10, 0x10, 0xc208444400, 0x0, 0x0)
    /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/godbus/dbus/transport_unix.go:21 +0xc5
io.ReadAtLeast(0x7f6ceef97e68, 0xc208444400, 0xc20843dc00, 0x10, 0x10, 0x10, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:298 +0xf1
io.ReadFull(0x7f6ceef97e68, 0xc208444400, 0xc20843dc00, 0x10, 0x10, 0x5c, 0x0, 0x0)
    /usr/local/go/src/io/io.go:316 +0x6d
github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc208418d60, 0xc208422750, 0x0, 0x0)
    /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/godbus/dbus/transport_unix.go:85 +0x1bf
github.com/godbus/dbus.(*Conn).inWorker(0xc2080ebd40)
    /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/godbus/dbus/conn.go:248 +0x58
created by github.com/godbus/dbus.(*Conn).Auth
    /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/godbus/dbus/auth.go:118 +0xe84

goroutine 38 [chan receive]:
github.com/godbus/dbus.(*Conn).outWorker(0xc2080ebd40)
    /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/godbus/dbus/conn.go:370 +0x58
created by github.com/godbus/dbus.(*Conn).Auth
    /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/godbus/dbus/auth.go:119 +0xea1

goroutine 39 [chan receive]:
github.com/coreos/go-systemd/dbus.func·001()
    /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/coreos/go-systemd/dbus/subscription.go:70 +0x64
created by github.com/coreos/go-systemd/dbus.(*Conn).initDispatch
    /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/coreos/go-systemd/dbus/subscription.go:94 +0x11c

goroutine 40 [chan receive]:
github.com/docker/docker/daemon.(*statsCollector).run(0xc208448b70)
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/stats_collector.go:89 +0xb2
created by github.com/docker/docker/daemon.newStatsCollector
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/stats_collector.go:29 +0x116

goroutine 41 [chan receive]:
github.com/docker/docker/pkg/signal.func·002()
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:29 +0x8f
created by github.com/docker/docker/pkg/signal.Trap
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:55 +0x250

goroutine 48 [IO wait]:
net.(*pollDesc).Wait(0xc208083170, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208083170, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208083110, 0xc208459000, 0x1000, 0x1000, 0x0, 0x7f6ceef8f1a0, 0xc208631298)
    /usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2080375f8, 0xc208459000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0xdc
crypto/tls.(*block).readFromUntil(0xc208465950, 0x7f6ceef98970, 0xc2080375f8, 0x5, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:454 +0xe6
crypto/tls.(*Conn).readRecord(0xc20807e580, 0x17, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:539 +0x2da
crypto/tls.(*Conn).Read(0xc20807e580, 0xc208450000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:904 +0x166
net/http.noteEOFReader.Read(0x7f6ceef9d480, 0xc20807e580, 0xc2084577b8, 0xc208450000, 0x1000, 0x1000, 0xae0660, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc208595cc0, 0xc208450000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0)
    <autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc20845aae0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc20845aae0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208457760)
    /usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 49 [select]:
net/http.(*persistConn).writeLoop(0xc208457760)
    /usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0xcbc

goroutine 91 [runnable]:
net/http.(*persistConn).readLoop(0xc208457550)
    /usr/local/go/src/net/http/transport.go:928 +0x9ce
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 67 [chan send]:
github.com/docker/docker/graph.func·009(0xc2083dd4a0)
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/graph/pull.go:626 +0x6b
created by github.com/docker/docker/graph.(*TagStore).pullV2Tag
    /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/graph/pull.go:627 +0x2671

goroutine 92 [select]:
net/http.(*persistConn).writeLoop(0xc208457550)
    /usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0xcbc

Docker info:

docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-253:0-136942298-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 0 B
 Data Space Total: 107.4 GB
 Data Space Available: 107.4 GB
 Metadata Space Used: 577.5 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 7.753 GiB
Name: localhost
ID: ZXRV:PWQW:RIZR:Z5QV:LJX3:7UUY:LA7L:UDQQ:D577:NH6B:SWUN:R57U
Username: eleet
Registry: https://index.docker.io/v1/

Kernel:

uname -a
Linux localhost 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

OS:

cat /etc/redhat-release 
CentOS Linux release 7.1.1503 (Core) 

Running on physical machine (localhost @ my room) with no virtualization.

Any help would be greatly appreciated!

EDIT:
With identical installation (docker, os, etc) on different machine with the only difference (I believe) is the Kernel it seems to work:

uname -a
Linux ip-10-0-112-146.ec2.internal 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Sep 22 19:06:58 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

EDIT2:
Getting the same error on this kernel:

uname -a
Linux m3210.contabo.host 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

The error seems pretty serious as it crashes the daemon with a panic: runtime error: invalid memory address or nil pointer dereference. Any pointers to what is causing this?

EDIT3:

./check-config.sh 
warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-3.10.0-229.el7.x86_64 ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_MACVLAN: enabled (as module)
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_MEMCG_KMEM: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
- CONFIG_RESOURCE_COUNTERS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_NETPRIO_CGROUP: enabled (as module)
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_EXT3_FS: missing
- CONFIG_EXT3_FS_XATTR: missing
- CONFIG_EXT3_FS_POSIX_ACL: missing
- CONFIG_EXT3_FS_SECURITY: missing
    (enable these ext3 configs if you are using ext3 as backing filesystem)
- CONFIG_EXT4_FS: enabled (as module)
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

EDIT4:
FIXED: rm -rf /var/lib/docker/ does it. Still think bad dms shouldn't cause a panic in the docker daemon though.

/b3

@beetree
beetree commented Aug 30, 2015

Seriously? A reproduceable Panic in the docker daemon and no reaction from the devs after 10 days?

Have they taken the money and left?

@VivaceVivo

I had the same problem and was not able to fix it because I did'nt want to go for the full nuke: /var/lib/docker/ solution.
I found something promising at:
http://grokbase.com/t/gg/docker-user/1563fzdtm7/docker-docker-runs-out-of-space-when-trying-to-create-a-new-image
'The default docker storage driver allocates 10GB storage blocks for your
images. Move to overlayfs and avoid this entirely. In the command that
starts your docker daemon just add "-s overlay" '

@ted-bunny

Had the same issue following the tutorial @ http://docs.docker.com/engine/installation/ubuntulinux/ on ubuntu trusty. But "sudo apt-get install linux-image-extra-uname -r" did the trick, thanks @francisdb !

@JorisBolsens

uname -a

Linux localhost.localdomain 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

docker version

Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 18:00:05 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 18:00:05 UTC 2015
 OS/Arch:      linux/amd64

docker info

Containers: 0
Images: 455
Server Version: 1.9.0
Storage Driver: devicemapper
 Pool Name: docker-253:0-2079720-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 42.23 GB
 Data Space Total: 107.4 GB
 Data Space Available: 4.793 GB
 Metadata Space Used: 59.08 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.088 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 3.703 GiB
Name: localhost.localdomain
ID: ZTLO:6OGG:NVMM:26PX:HF2A:ZGA2:Y3ES:UIWR:L3M6:CSZI:KAWJ:IZI4
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
@yegor256 yegor256 referenced this issue in yegor256/rultor Dec 7, 2015
Closed

Build pipeline is broken #959

@everhopingandwaiting

version is :

root@AY131031171703122b59Z:~# docker version 
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:08:59 UTC 2015
 OS/Arch:      linux/amd64
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
root@AY131031171703122b59Z:~# 

what's wrong with "Cannot connect to the Docker daemon. Is the docker daemon running on this host?" ?

root@AY131031171703122b59Z:~# docker -d -D
Warning: '-d' is deprecated, it will be removed soon. See usage.
WARN[0000] please use 'docker daemon' instead.          
DEBU[0000] docker group found. gid: 999                 
DEBU[0000] Server created for HTTP on unix (/var/run/docker.sock) 
WARN[0000] Your Linux kernel version 3.2.0-97-generic can be unstable running docker. Please upgrade your kernel to 3.10.0. 
INFO[0000] API listen on /var/run/docker.sock           
INFO[0000] [graphdriver] using prior storage driver "aufs" 
DEBU[0000] Using graph driver aufs                      
DEBU[0000] Using default logging driver json-file       
DEBU[0000] Migrating existing containers                
DEBU[0000] Creating images graph                        
DEBU[0000] Restored 0 elements                          
DEBU[0000] Creating repository list                     
DEBU[0000] Option DefaultDriver: bridge                 
DEBU[0000] Option DefaultNetwork: bridge                
INFO[0000] Firewalld running: false                     
DEBU[0000] /sbin/iptables, [-t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[0000] /sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER] 
DEBU[0000] /sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[0000] /sbin/iptables, [-t nat -D PREROUTING]       
DEBU[0000] /sbin/iptables, [-t nat -D OUTPUT]           
DEBU[0000] /sbin/iptables, [-t nat -F DOCKER]           
DEBU[0000] /sbin/iptables, [-t nat -X DOCKER]           
DEBU[0000] /sbin/iptables, [-t nat -n -L DOCKER]        
DEBU[0000] /sbin/iptables, [-t nat -N DOCKER]           
DEBU[0000] /sbin/iptables, [-t filter -n -L DOCKER]     
DEBU[0000] Registering ipam provider: default           
DEBU[0000] Allocating IPv4 pools for network bridge (2247bf51bd9593c17fbc812ecd3c1363ed0584b0a93dd77f14c821af20fcaa9f) 
DEBU[0000] RequestPool(LocalDefault, , , map[], false)  
DEBU[0000] Cleaning up old shm/mqueue mounts: start.    
FATA[0000] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network 
root@AY131031171703122b59Z:~# 

ubuntu kernel version is :

root@AY131031171703122b59Z:~# uname -a
Linux AY131031171703122b59Z 3.2.0-97-generic #137-Ubuntu SMP Thu Dec 17 18:11:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@AY131031171703122b59Z:~# 

@everhopingandwaiting

---ok , I konw it is that my kernel is too lower,---

root@AY131031171703122b59Z:~# docker daemon
INFO[0000] API listen on /var/run/docker.sock           
INFO[0000] [graphdriver] using prior storage driver "aufs" 
INFO[0000] Firewalld running: false                     
FATA[0000] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network 
root@AY131031171703122b59Z:~# uname -a
Linux AY131031171703122b59Z 3.13.0-74-generic #118~precise1-Ubuntu SMP Fri Dec 18 10:38:55 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@AY131031171703122b59Z:~# 
@andyneff
andyneff commented Feb 12, 2016 edited

TL;DR

I'm constantly corrupting my /var/lib/docker directory by running docker in loopback mode when I MEAN to run in devicemapper mode.

To fix without rebooting/deleting the ENTIRE /var/lib/docker dir (as root)

  1. dmsetup ls # list docker-* devices
  2. dmsetup remove docker-* #fill in the * yourself, unmaps loopback device
  3. rm -rv /var/lib/docker/devicemapper/devicemapper
  4. Restart docker in devicemapper mode

Explanation

I'm not sure if this is of any help, but I am CONTINUOUSLY having this problem too!

...
DEBU[0000] devmapper: Added deviceId=149 to DeviceIdMap 
DEBU[0000] devmapper: Added deviceId=226 to DeviceIdMap 
DEBU[0000] devmapper: constructDeviceIDMap() END        
DEBU[0000] devmapper: activateDeviceIfNeeded()          
DEBU[0000] devmapper: Error device setupBaseImage: devmapper: Base Device UUID and Filesystem verification failed.devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed 
DEBU[0000] Cleaning up old shm/mqueue mounts: start.    
FATA[0000] Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed.devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed 

Now the reason I get this is VERY reproducible and fixable. (NO, NOT BY DELETING /var/lib/docker!!!)

In my case, I run docker via

/usr/bin/docker daemon -H fd:// --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.basesize=100G

Which I add to /usr/lib/systemd/system/docker.service, HOWEVER every time I upgrade, that file get reset, and I inevitable forget to change it (AGAIN!) before trying to start docker... And here's the kick, when you start docker daemon without these options (in Centos 7), it will start in loopback mode or whatever. And when THAT happens, /var/lib/docker/devicemapper/devicemapper/ is populated. So when you DO run with the --storage-opt dm option, NOW the directory is corrupted. However, what I just discovered is that by deleting the /var/lib/docker/devicemapper/devicemapper/ directory ONLY, the problem is fixed. I'm able to start docker with the dm storage and keep all my images and containers..

Edit: However I have discovered that it is important to clean up the loop back devices properly BEFORE deleting /var/lib/docker/devicemapper/devicemapper/, or else you have stray loopbacks that need to be cleaned up (via rebooting?)

Recap

  1. /usr/bin/docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.basesize=100G #works

  2. /usr/bin/docker daemon #corrupts

  3. /usr/bin/docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.basesize=100G #broken

  4. dmsetup ls to list the mapped docker devices

    vg--docker-data (253:2)
    vg--docker-metadata (253:3)
    docker-253:1-2152645735-pool    (253:4)
    centos-swap (253:0)
    centos-root (253:1)
    
  5. dmsetup remove docker-253:1-2152645735-pool # This should remove the map on the loopback device AND remove the loopback devices, since autoclear is usually on, the vgs are something else, and belong there

  6. losetup -a #lists loopback devices (Should be empty of docker loopbacks)

  7. losetup -d /dev/loop0 # to unmount loopback device IF any are left. only unmount the ones you see in losetup -a, which might not be loop0

  8. rm -rv /var/lib/docker/devicemapper/devicemapper #fixes everything

  9. /usr/bin/docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.basesize=100G #works again

Running CentOS 7.2.1511

docker version

Client:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:34:50 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:34:50 2016
 OS/Arch:      linux/amd64

uname -a

Linux newbie 3.10.0-327.4.5.el7.x86_64 #1 SMP Mon Jan 25 22:07:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

As a side note, the third time I tried this, I had to delete deviceset-metadata and transaction-metadata from the /var/lib/docker/devicemapper/metadata directory. This may be unrelated, but the values got messed up, and deleting them lets them regenerate safely.

@chrislovecnm

Any guidelines on figuring this out??

@zgoda-mobica
# kill -9 $(lsof -t -c docker)
# rm -rf /var/lib/docker/*
# reboot

Works for me.

@firegate666

Hi, installed docker on my wheezy and have the same problem, the service does not start

Linux s17367650 3.10.32-20140228a #1 SMP Fri Feb 28 08:17:48 UTC 2014 x86_64 GNU/Linux

Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:13:43 2016
OS/Arch: linux/amd64

And this is what the logfile says:

time="2016-09-13T04:13:13.479620266+02:00" level=info msg="libcontainerd: new containerd process, pid: 12201"
time="2016-09-13T04:13:14.516135282+02:00" level=error msg="[graphdriver] prior storage driver "devicemapper" failed: devicemapper: Error running deviceCreate (CreatePool) dm_task_run failed"
time="2016-09-13T04:13:14.516264554+02:00" level=fatal msg="Error starting daemon: error initializing graphdriver: devicemapper: Error running deviceCreate (CreatePool) dm_task_run failed"

I followed this installation guide: https://docs.docker.com/engine/installation/linux/debian/

@motymichaely

Hey,

I am hitting this now.
Docker info:

Containers: 30
 Running: 16
 Paused: 0
 Stopped: 14
Images: 10
Server Version: 1.12.3
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 3.299 GB
 Data Space Total: 25.5 GB
 Data Space Available: 22.2 GB
 Metadata Space Used: 233.5 kB
 Metadata Space Total: 264.2 MB
 Metadata Space Available: 264 MB
 Thin Pool Minimum Free Space: 2.55 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-09-28)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host null bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.18.25-19.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 29.5 GiB
Name: ip-192-168-3-114
ID: YTNR:MPYN:UJAL:PJT5:KLBS:MJ62:6ZKD:P6MS:UH2H:G2QC:RZLZ:4MQF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

These are the results of check-config.sh:

info: reading kernel config from /boot/config-3.18.25-19.el7.x86_64 ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled

Optional Features:
- CONFIG_USER_NS: enabled
  (RHEL7/CentOS7: User namespaces disabled; add 'user_namespace.enable=1' to boot command line)
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: missing
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: missing
    (note that cgroup swap accounting is not enabled in your kernel config, you can enable it by setting boot option "swapaccount=1")
- CONFIG_MEMCG_KMEM: enabled
- CONFIG_RESOURCE_COUNTERS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT3_FS: enabled (as module)
- CONFIG_EXT3_FS_XATTR: enabled
- CONFIG_EXT3_FS_POSIX_ACL: enabled
- CONFIG_EXT3_FS_SECURITY: enabled
- CONFIG_EXT4_FS: enabled (as module)
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled (as module)
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

Any idea what's causing these corruptions? We are hitting this quite a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment