docker fails to mount the block device for the container on devicemapper #4036

Closed
unclejack opened this Issue Feb 10, 2014 · 391 comments

Projects

None yet
@unclejack
Contributor

When running something like for i in {0..100}; do docker run busybox echo test; done with Docker running on devicemapper, errors are thrown and containers fail to run:

2014/02/10 9:48:42 Error: start: Cannot start container 56bee8c4da5bd5641fc42405c742083b418ca14ddfb4a3e632955e236e23c284: Error getting container 56bee8c4da5bd5641fc42405c742083b418ca14ddfb4a3e632955e236e23c284 from driver devicemapper: Error mounting '/dev/mapper/docker-8:1-4980769-56bee8c4da5bd5641fc42405c742083b418ca14ddfb4a3e632955e236e23c284' on '/var/lib/docker/devicemapper/mnt/56bee8c4da5bd5641fc42405c742083b418ca14ddfb4a3e632955e236e23c284': no such file or directory
2014/02/10 9:48:42 Error: start: Cannot start container b90b4385778142aab5251846460008e5c4eb9fe1e7ec82f07d06f1de823bd914: Error getting container b90b4385778142aab5251846460008e5c4eb9fe1e7ec82f07d06f1de823bd914 from driver devicemapper: Error mounting '/dev/mapper/docker-8:1-4980769-b90b4385778142aab5251846460008e5c4eb9fe1e7ec82f07d06f1de823bd914' on '/var/lib/docker/devicemapper/mnt/b90b4385778142aab5251846460008e5c4eb9fe1e7ec82f07d06f1de823bd914': no such file or directory
2014/02/10 9:48:43 Error: start: Cannot start container ca53b3b21c92ffb17ad15c1088be293260ea240abdf25db7e5aadc11517cf93c: Error getting container ca53b3b21c92ffb17ad15c1088be293260ea240abdf25db7e5aadc11517cf93c from driver devicemapper: Error mounting '/dev/mapper/docker-8:1-4980769-ca53b3b21c92ffb17ad15c1088be293260ea240abdf25db7e5aadc11517cf93c' on '/var/lib/docker/devicemapper/mnt/ca53b3b21c92ffb17ad15c1088be293260ea240abdf25db7e5aadc11517cf93c': no such file or directory
test
2014/02/10 9:48:43 Error: start: Cannot start container 1e1e06044711e73cede8ede10547de7e270c33fac7ad5e60a8cb23246950adf3: Error getting container 1e1e06044711e73cede8ede10547de7e270c33fac7ad5e60a8cb23246950adf3 from driver devicemapper: Error mounting '/dev/mapper/docker-8:1-4980769-1e1e06044711e73cede8ede10547de7e270c33fac7ad5e60a8cb23246950adf3' on '/var/lib/docker/devicemapper/mnt/1e1e06044711e73cede8ede10547de7e270c33fac7ad5e60a8cb23246950adf3': no such file or directory

Fedora 20 with kernel 3.12.9 doesn't seem to be affected.

kernel version, distribution, docker info and docker version:

3.11.0-15-generic #25~precise1-Ubuntu SMP Thu Jan 30 17:39:31 UTC 2014 x86_64 x86_64
Ubuntu 12.04.4
 docker info
Containers: 101
Images: 44
Driver: devicemapper
 Pool Name: docker-8:1-4980769-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 3234.9 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 6.9 Mb
 Metadata Space Total: 2048.0 Mb

Client version: 0.8.0-dev
Go version (client): go1.2
Git commit (client): 695719b
Server version: 0.8.0-dev
Git commit (server): 695719b
Go version (server): go1.2
Last stable version: 0.8.0

The Docker binary is actually master with PR #4017 merged.

/cc @alexlarsson

@discordianfish
Contributor

Same here, running ubuntu precise with lts-raring kernel (3.8.0-35-generic).

@alexlarsson
Contributor

This is due to a race with udev. The problem is that starting a container creates a dm device, activates it and then immediately deactivates it and then activates it again. This races with the udev device-node creation such that you end up in a state where the device node created by udev is removed by docker.

@alexlarsson alexlarsson added a commit to alexlarsson/docker that referenced this issue Feb 11, 2014
@alexlarsson alexlarsson Avoid extra mount/unmount during container registration
Runtime.Register() called driver.Get()/Put() in order to read back the
basefs of the container. However, this is not needed, as the basefs
is read during container.Mount() anyway, and basefs is only valid
while mounted (and all current calls satisfy this).

This seems minor, but this is actually problematic, as the Get/Put
pair will create a spurious mount/unmount cycle that is not needed and
slows things down. Additionally it will create a supurious
devicemapper activate/deactivate cycle that causes races with udev as
seen in docker#4036.

With this change devicemapper is now race-free, and container startup
is slightly faster.

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
0c71015
@unclejack
Contributor

@alexlarsson Could we work around it somehow? It looks like every non-Fedora system has this problem.

@alexlarsson
Contributor

@unclejack #4067 fixes it

@discordianfish
Contributor

The problem seems to be fixed for runs, but not for builds. When building this Dockerfile:

FROM busybox
RUN  echo hello world

docker fails with the same error:

2014/02/12 03:30:52 build: Error getting container c9bfa29eaa875796895e6e1994b5437d983833da3249d4ea3337e3f560fab5af from driver devicemapper: Error mounting '/dev/mapper/docker-252:0-5767278-c9bfa29eaa875796895e6e1994b5437d983833da3249d4ea3337e3f560fab5af' on '/var/lib/docker/devicemapper/mnt/c9bfa29eaa875796895e6e1994b5437d983833da3249d4ea3337e3f560fab5af': no such file or directory
@alexlarsson
Contributor

@discordianfish I'll have a look at that.

@alexlarsson alexlarsson added a commit to alexlarsson/docker that referenced this issue Feb 12, 2014
@alexlarsson alexlarsson Avoid extra mount/unmount during build
CmdRun() calls first run() and then wait() to wait for it to exit,
then it runs commit(). The run command will mount the container and
the container exiting will unmount it. Then the commit will
immediately mount it again to do a diff.

This seems minor, but this is actually problematic, as the Get/Put
pair will create a spurious mount/unmount cycle that is not needed and
slows things down. Additionally it will create a supurious
devicemapper activate/deactivate cycle that causes races with udev as
seen in docker#4036.

To ensure that we only unmount once we split up run() into create()
and run() and reference the mount until after the commit().

With this change docker build on devicemapper is now race-free, and
slightly faster.

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
59347fa
@alexlarsson
Contributor

#4096 fixes a similar spurious mount/unmount cycle during build.

@alexlarsson
Contributor

While the above fixes will fix this for most people we should probably leave this open to track the actual problem of the udev race, as it would be good to have a fix for that too.

@discordianfish
Contributor

The above fixes are a big improvement, but when restarting container it sometimes still happens:

docker ps -q|xargs docker restart
Error: restart: Cannot restart container 3b96cdfee1eb: Error getting container 3b96cdfee1eb5e5b70d94d9a30e0c1cf9e9eba83901d63fd3e509df81cef8627 from driver devicemapper: Error mounting '/dev/mapper/docker-252:0-264246-3b96cdfee1eb5e5b70d94d9a30e0c1cf9e9eba83901d63fd3e509df81cef8627' on '/var/lib/docker/devicemapper/mnt/3b96cdfee1eb5e5b70d94d9a30e0c1cf9e9eba83901d63fd3e509df81cef8627': no such file or directory
Error: restart: Cannot restart container d968669da4ce: Error getting container d968669da4ce5632416e55fac56666c588e8ed236874394e9bbfa5e3561acb81 from driver devicemapper: Error mounting '/dev/mapper/docker-252:0-264246-d968669da4ce5632416e55fac56666c588e8ed236874394e9bbfa5e3561acb81' on '/var/lib/docker/devicemapper/mnt/d968669da4ce5632416e55fac56666c588e8ed236874394e9bbfa5e3561acb81': no such file or directory
96fd55c11524
6c0860bb723d
@unclejack unclejack added a commit to unclejack/docker that referenced this issue Feb 14, 2014
@alexlarsson @unclejack alexlarsson + unclejack Avoid extra mount/unmount during build
CmdRun() calls first run() and then wait() to wait for it to exit,
then it runs commit(). The run command will mount the container and
the container exiting will unmount it. Then the commit will
immediately mount it again to do a diff.

This seems minor, but this is actually problematic, as the Get/Put
pair will create a spurious mount/unmount cycle that is not needed and
slows things down. Additionally it will create a supurious
devicemapper activate/deactivate cycle that causes races with udev as
seen in docker#4036.

To ensure that we only unmount once we split up run() into create()
and run() and reference the mount until after the commit().

With this change docker build on devicemapper is now race-free, and
slightly faster.

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
904d444
@unclejack unclejack added a commit to unclejack/docker that referenced this issue Feb 14, 2014
@alexlarsson @unclejack alexlarsson + unclejack Avoid extra mount/unmount during container registration
Runtime.Register() called driver.Get()/Put() in order to read back the
basefs of the container. However, this is not needed, as the basefs
is read during container.Mount() anyway, and basefs is only valid
while mounted (and all current calls satisfy this).

This seems minor, but this is actually problematic, as the Get/Put
pair will create a spurious mount/unmount cycle that is not needed and
slows things down. Additionally it will create a supurious
devicemapper activate/deactivate cycle that causes races with udev as
seen in docker#4036.

With this change devicemapper is now race-free, and container startup
is slightly faster.

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
acc2ea2
@crosbymichael crosbymichael pushed a commit to crosbymichael/docker that referenced this issue Feb 15, 2014
@alexlarsson alexlarsson + Michael Crosby Avoid extra mount/unmount during container registration
Runtime.Register() called driver.Get()/Put() in order to read back the
basefs of the container. However, this is not needed, as the basefs
is read during container.Mount() anyway, and basefs is only valid
while mounted (and all current calls satisfy this).

This seems minor, but this is actually problematic, as the Get/Put
pair will create a spurious mount/unmount cycle that is not needed and
slows things down. Additionally it will create a supurious
devicemapper activate/deactivate cycle that causes races with udev as
seen in docker#4036.

With this change devicemapper is now race-free, and container startup
is slightly faster.

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
b19bdb7
@alexlarsson alexlarsson added a commit to alexlarsson/docker that referenced this issue Feb 17, 2014
@alexlarsson alexlarsson Avoid temporarily unmounting the container when restarting it
Stopping the container will typicall cause it to unmount, to keep it mounted
over the stop/start cycle we aquire a temporary reference to it during this time.

This helps with docker#4036

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
ab0f3f8
@alexlarsson
Contributor

@discordianfish The restart issue should be fixed in #4180

@unclejack
Contributor

These issues aren't occurring any more on any of my test systems.

@unclejack unclejack closed this Feb 21, 2014
@alexlarsson
Contributor

I believe it is still possible to run into the race if you e.g. do:
docker wait $id
docker diff $if
or something, where the container exits and we immediately start an operation that mounts it.
However, the race is pretty small, so maybe its not possible in practice...

@alexlarsson
Contributor

Short description of what I believe is happening:

fish_: its a lot more complex
fish_: both udev and docker are creating the nodes, and if this happens at the same time it gets into a weird situation where the device is activated but the device node doesn't exist
something like:
udev creates node
we see it, so don't create
we activate
udev activates
udev looks at the device, runs rules
udev removes node
=> we're in a state where we activated and created the node, but the node is now gone

@discordianfish
Contributor

Reopening still the actual problem is still there.

@discordianfish discordianfish reopened this Mar 4, 2014
@shykes shykes added this to the 0.9.1 milestone Mar 10, 2014
@shykes
Contributor
shykes commented Mar 10, 2014

Tentatively scheduling for 0.9.1

@unclejack
Contributor

There's still a problem with devicemapper.
Using the script:

for i in {1..50}
do

CID=`docker run -d -P registry`
for j in {1..20}
do
docker kill $CID 2> /dev/null
docker restart $CID
done

docker kill $CID 2> /dev/null
docker rm $CID 2> /dev/null &

done

Docker ends up throwing an error which looks like this:

Error: Cannot restart container 3bb6ce8fa5b76b6c6c8bd2adcb20e473ca9b07b78d425a4dper/docker-8:1-1052351-3bb6ce8fa5b76b6c6c8bd2adcb20e473ca9b07b78d425a4dafcbccd66
2014/03/10 20:04:28 Error: failed to restart one or more containers

The environments on which this occurs are both running Ubuntu precise:

Ubuntu 12.04.4 LTS
Linux ubu1204-3 3.8.0-36-generic #52~precise1-Ubuntu SMP Mon Feb 3 21:54:46 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Linux ubu1204-2 3.11.0-17-generic #31~precise1-Ubuntu SMP Tue Feb 4 21:25:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

This is a problem on 0.8.1 and it's still a problem on 0.9.0.
@alexlarsson Could this be the same udev race condition?

@alexlarsson
Contributor

We could try a workaround where we gratuitously keep a ref on a recently created device (unless you're trying to delete it) for say 1 second.

@discordianfish
Contributor

I ran into another error which might or might not be related:

$ docker rm 5da32d8cfd63
Error: container_delete: Cannot destroy container 5da32d8cfd63: Driver devicemapper failed to remove root filesystem 5da32d8cfd63e59ad6f2dfdc189bdc62f3e25e8b6d90ac5407e485ce610e9001: UnmountDevice: no such device 5da32d8cfd63e59ad6f2dfdc189bdc62f3e25e8b6d90ac5407e485ce610e9001
2014/03/18 15:04:14 Error: failed to remove one or more containers

(Container is stopped for some weeks already, system was rebooted since that several times)

@alexlarsson
Contributor
@alexlarsson
Contributor
@creack
Contributor
creack commented Mar 18, 2014

The timeout PR seems to have solve the issue /cc @unclejack

@rws-github

I'm still getting the error even with the changes:

2014/03/21 17:05:37 Error mounting '/dev/mapper/docker-8:1-2138150-e85daf5656ab5bc43aa7153f21f9dd5b54473cada547f00763c9e89b28360e33-init' on '/var/lib/docker/devicemapper/mnt/e85daf5656ab5bc43aa7153f21f9dd5b54473cada547f00763c9e89b28360e33-init': invalid argument

@discordianfish
Contributor

@rws-github This looks different. You get 'invalid argument' where I got 'no such file or directory'.

@XiaZhangBJ

I am also encounted the same problem same as rws-github's, anyone can give me a help?

[root@localhost ~]# docker run busybox /bin/echo hello world
2014/03/26 11:10:03 Error: create: Error mounting '/dev/mapper/docker-8:6-368797-bdaad691584b742d519f27d7762b686bf4e1c998e06eddac805e69514758ad31-init' on '/var/lib/docker/devicemapper/mnt/bdaad691584b742d519f27d7762b686bf4e1c998e06eddac805e69514758ad31-init': invalid argument

demon (docker -d) shown as below:
2014/03/26 11:10:03 POST /v1.9/containers/create
[/var/lib/docker|4172e75d] +job create()
Error mounting '/dev/mapper/docker-8:6-368797-bdaad691584b742d519f27d7762b686bf4e1c998e06eddac805e69514758ad31-init' on '/var/lib/docker/devicemapper/mnt/bdaad691584b742d519f27d7762b686bf4e1c998e06eddac805e69514758ad31-init': invalid argument[/var/lib/docker|4172e75d] -job create() = ERR (1)
[error] api.go:998 Error: create: Error mounting '/dev/mapper/docker-8:6-368797-bdaad691584b742d519f27d7762b686bf4e1c998e06eddac805e69514758ad31-init' on '/var/lib/docker/devicemapper/mnt/bdaad691584b742d519f27d7762b686bf4e1c998e06eddac805e69514758ad31-init': invalid argument
[error] api.go:105 HTTP Error: statusCode=500 create: Error mounting '/dev/mapper/docker-8:6-368797-bdaad691584b742d519f27d7762b686bf4e1c998e06eddac805e69514758ad31-init' on '/var/lib/docker/devicemapper/mnt/bdaad691584b742d519f27d7762b686bf4e1c998e06eddac805e69514758ad31-init': invalid argument

[root@localhost ~]# docker info
Containers: 0
Images: 5
Driver: devicemapper
Pool Name: docker-8:6-368797-pool
Data file: /var/lib/docker/devicemapper/devicemapper/data
Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 38.1 Mb
Data Space Total: 102400.0 Mb
Metadata Space Used: 0.9 Mb
Metadata Space Total: 2048.0 Mb

[root@localhost ~]# uname -a
Linux localhost.localdomain 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# cat /etc/redhat-release
CentOS release 6.4 (Final)

@rageshkrishna

Same invalid argument error here. I had > 200 containers on a machine that had to be hard reset. The containers did not come back up automatically, and trying docker start on them would give me this error message. This is with v0.9.1 on Ubuntu 13.10, kernel 3.11.0-15-generic.

@unclejack
Contributor

@rageshkrishna Docker 0.9.1 isn't the latest any more. Please update it to 0.10.0 and report back if you are still having these problems.

@rageshkrishna

@unclejack I'm still seeing this with Docker 0.10.0. This is on the same machine. Upgraded from 0.9.1 to 0.10.0, removed all existing containers and images and started fresh. Pulled an image from my registry, fired up a container, then did docker stop and docker start on it.

Docker logs:

2014/04/19 13:12:24 POST /v1.10/containers/ceda/stop?t=10
[/data|80d2093a] +job stop(ceda)
2014/04/19 13:12:34 Container ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e failed to exit within 10 seconds of SIGTERM - using the force
[/data|80d2093a] +job release_interface(ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e)
2014/04/19 13:12:34 Stopping proxy on tcp/[::]:49153 for tcp/172.17.0.2:22 (accept tcp [::]:49153: use of closed network connection)
[/data|80d2093a] -job release_interface(ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e) = OK (0)
[error] container.go:872 ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e: Error closing terminal: invalid argument
[/data|80d2093a] -job stop(ceda) = OK (0)
2014/04/19 13:12:38 POST /v1.10/containers/ceda/start
[/data|80d2093a] +job start(ceda)
[/data|80d2093a] +job release_interface(ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e)
2014/04/19 13:12:38 Unable to unmap port 0.0.0.0:49153: port is not mapped
[/data|80d2093a] -job release_interface(ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e) = OK (0)
[error] container.go:872 ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e: Error closing terminal: invalid argument
[error] driver.go:127 Warning: error unmounting device ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e: UnmountDevice: device not-mounted id ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e


Cannot start container ceda: Error getting container ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e from driver devicemapper: Error mounting '/dev/mapper/docker-202:32-43515905-ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e' on '/data/devicemapper/mnt/ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e': invalid argument
[/data|80d2093a] -job start(ceda) = ERR (1)
[error] server.go:962 Error: Cannot start container ceda: Error getting container ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e from driver devicemapper: Error mounting '/dev/mapper/docker-202:32-43515905-ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e' on '/data/devicemapper/mnt/ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e': invalid argument
[error] server.go:89 HTTP Error: statusCode=500 Cannot start container ceda: Error getting container ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e from driver devicemapper: Error mounting '/dev/mapper/docker-202:32-43515905-ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e' on '/data/devicemapper/mnt/ceda5a44a6f08b8af1403eee8ac8382e48b9f1af9d45ba8a9e8c0232a8b2ca8e': invalid argument

@unclejack
Contributor

@rageshkrishna Did you also reboot the machine after upgrading to Docker 0.10?

@rageshkrishna

@unclejack Yes, I had no choice but to reboot. After the upgrade the old docker daemon was still running. I couldn't stop it, and it wouldn't let me launch the new one.

@vieira
vieira commented Apr 20, 2014

I think this problem exists in Ubuntu 14.04 with docker 0.10:

# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
889c186264ca        artm/dev:latest     /usr/bin/init-dev   5 seconds ago       Up 5 seconds        127.0.0.1:1002->80/tcp   trusting_engelbart   
# stop docker
docker stop/waiting
# start docker
docker start/running, process 1375
# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS                     PORTS                    NAMES
889c186264ca        artm/dev:latest     /usr/bin/init-dev   About a minute ago   Exited (0) 4 seconds ago   127.0.0.1:1002->80/tcp   trusting_engelbart   
# docker start 889c186264ca
Error: Cannot start container 889c186264ca: Error getting container 889c186264ca0d4095153efdf48d1816ab62762d58702565437799d7f2d73522 from driver devicemapper: Error mounting '/dev/mapper/docker-253:0-267081-889c186264ca0d4095153efdf48d1816ab62762d58702565437799d7f2d73522' on '/var/lib/docker/devicemapper/mnt/889c186264ca0d4095153efdf48d1816ab62762d58702565437799d7f2d73522': device or resource busy
2014/04/20 00:55:15 Error: failed to start one or more containers
# docker rm 889c186264ca
Error: Cannot destroy container 889c186264ca: Driver devicemapper failed to remove root filesystem 889c186264ca0d4095153efdf48d1816ab62762d58702565437799d7f2d73522: Error running removeDevice
2014/04/20 01:02:15 Error: failed to remove one or more containers

# docker version
Client version: 0.10.0
Client API version: 1.10
Go version (client): go1.2.1
Git commit (client): dc9c28f
Server version: 0.10.0
Server API version: 1.10
Git commit (server): dc9c28f
Go version (server): go1.2.1
Last stable version: 0.10.0

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04 LTS
Release:    14.04
Codename:   trusty

# uname -a
Linux dev.yubo.be 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
@crosbymichael crosbymichael modified the milestone: 1.0, 0.9.1 May 15, 2014
@crosbymichael crosbymichael removed this from the 1.0 milestone May 30, 2014
@tpires
tpires commented Jun 9, 2014

Having this problem on RH 6.5 with docker 0.11.1 (only reboot solved the problem):

# docker ps -a
CONTAINER ID        IMAGE                                  COMMAND                CREATED             STATUS                      PORTS                                                                                                                                                                                            NAMES
e7095613a46c        shipyard/shipyard:latest               /app/.docker/run.sh    7 weeks ago         Exited (0) 21 minutes ago                                                                                                                                                                                                    shipyard                                                                                             
8577cef2f803        shipyard/db:latest                     /bin/bash -e /usr/lo   7 weeks ago         Exited (0) 21 minutes ago                                                                                                                                                                                                    shipyard/db,shipyard_db                                                                              
aa9996cf015e        shipyard/lb:latest                     /bin/sh -e /usr/loca   7 weeks ago         Exited (0) 21 minutes ago                                                                                                                                                                                                    shipyard_lb                                                                                          
631e3f7d2d1f        shipyard/router:latest                 /bin/sh -e /usr/loca   7 weeks ago         Exited (0) 21 minutes ago                                                                                                                                                                                                    shipyard_lb/app_router,shipyard_router                                                               
091a01f48f30        shipyard/redis:latest                  /usr/local/bin/redis   7 weeks ago         Up 4 minutes                0.0.0.0:6379->6379/tcp                                                                                                                                                                           shipyard/redis,shipyard_lb/app_router/redis,shipyard_lb/redis,shipyard_redis,shipyard_router/redis   
263be244cf6c

# docker start 8577cef2f803 631e3f7d2d1f aa9996cf015e e7095613a46c
Error: Cannot start container 8577cef2f803: Error getting container 8577cef2f8038d0565a8da9a7d1f12d49f9b0673610788b941eaeca686b6b70d from driver devicemapper: Error mounting '/dev/mapper/docker-253:1-393380-8577cef2f8038d0565a8da9a7d1f12d49f9b0673610788b941eaeca686b6b70d' on '/var/lib/docker/devicemapper/mnt/8577cef2f8038d0565a8da9a7d1f12d49f9b0673610788b941eaeca686b6b70d': device or resource busy
Error: Cannot start container 631e3f7d2d1f: Error getting container 631e3f7d2d1f57ebf78b77347c9c8567c44247557264a742d6e655f933dc4c55 from driver devicemapper: Error mounting '/dev/mapper/docker-253:1-393380-631e3f7d2d1f57ebf78b77347c9c8567c44247557264a742d6e655f933dc4c55' on '/var/lib/docker/devicemapper/mnt/631e3f7d2d1f57ebf78b77347c9c8567c44247557264a742d6e655f933dc4c55': device or resource busy
Error: Cannot start container aa9996cf015e: Error getting container aa9996cf015e484ecc59575adc12b2bbf04d4761dafac632abcd66fede80ada3 from driver devicemapper: Error mounting '/dev/mapper/docker-253:1-393380-aa9996cf015e484ecc59575adc12b2bbf04d4761dafac632abcd66fede80ada3' on '/var/lib/docker/devicemapper/mnt/aa9996cf015e484ecc59575adc12b2bbf04d4761dafac632abcd66fede80ada3': device or resource busy
Error: Cannot start container e7095613a46c: Error getting container e7095613a46c270c38b3cef4ba7927187727a238faa898d585064c0d3981eaae from driver devicemapper: Error mounting '/dev/mapper/docker-253:1-393380-e7095613a46c270c38b3cef4ba7927187727a238faa898d585064c0d3981eaae' on '/var/lib/docker/devicemapper/mnt/e7095613a46c270c38b3cef4ba7927187727a238faa898d585064c0d3981eaae': device or resource busy
2014/06/09 15:28:28 Error: failed to start one or more containers

# docker version
Client version: 0.11.1
Client API version: 1.11
Go version (client): go1.2.1
Git commit (client): fb99f99/0.11.1
Server version: 0.11.1
Server API version: 1.11
Git commit (server): fb99f99/0.11.1
Go version (server): go1.2.1

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.5 (Santiago)

# uname -a
Linux vm-host 2.6.32-431.17.1.el6.x86_64 #1 SMP Fri Apr 11 17:27:00 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

To reproduce this error i just need to:

# service docker restart
@fcuozzo
fcuozzo commented Jun 10, 2014

Got the error while doing a 'docker build'

---> Running in f76a1c31aea7
2014/06/10 02:25:09 Error getting container init rootfs f76a1c31aea7e6f097450ab9e8b10faa70e517a19b0fb53768d7e607592eb3b2 from driver devicemapper: Error mounting '/dev/mapper/docker-202:1-20763-f76a1c31aea7e6f097450ab9e8b10faa70e517a19b0fb53768d7e607592eb3b2-init' on '/var/lib/docker/devicemapper/mnt/f76a1c31aea7e6f097450ab9e8b10faa70e517a19b0fb53768d7e607592eb3b2-init': no such file or directory
# docker version
Client version: 1.0.0
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 63fe64c
Server version: 1.0.0
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 63fe64c

This seems to be some sort of racing condition, we were seeing the same issue with 0.11.1 in scenarios similar to the described above the others.

# uname -a
#50-Ubuntu SMP Thu May 15 18:06:16 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04 LTS
Release:    14.04
Codename:   trusty

Anything we can do to help diagnose/fix this?

@janstenpickle

I can confirm this is happening for me in 1.0 also. I'm building a lot of containers so hit this race condition in every build, I used to work around it by using AUFS, while still in the codebase it appears that the driver has been disabled in the 1.0 builds of docker.

Error getting container init rootfs 85f54472a5861e01c68f1fafe40be9d573fdec4c9cba8d350ba6795d054d6dca from driver devicemapper: Error mounting '/dev/mapper/docker-202:16-8159234-85f54472a5861e01c68f1fafe40be9d573fdec4c9cba8d350ba6795d054d6dca-init' on '/mnt/docker/devicemapper/mnt/85f54472a5861e01c68f1fafe40be9d573fdec4c9cba8d350ba6795d054d6dca-init': no such file or directory
#docker version
Client version: 1.0.0
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 63fe64c
Server version: 1.0.0
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 63fe64c
# uname -a
Linux dev 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04 LTS
Release:    14.04
Codename:   trusty
@kuro5hin

I get this intermittently during container builds too.

$ docker version
Client version: 1.0.0
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 63fe64c
Server version: 1.0.0
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 63fe64c

$ uname -a
Linux dev 3.11.0-23-generic #40~precise1-Ubuntu SMP Wed Jun 4 22:06:36 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.4 LTS
Release: 12.04
Codename: precise

@danbeaulieu

Also experiencing this with the same exact environment as comment #4036 (comment)

@danielschonfeld

Same here on ec2 building containers of various sized on an EBS block device. Ubuntu 14.04 + Docker 1.0.1

@garlandkr

Similar setup to others, Ubuntu 14.04, Docker 1.0.1 - running a build I get _Error getting container init rootfs_

One thing I'd like to add is the output from ps -a shows this command running:

 /bin/sh -c '#(nop) A

This happens for all of the different builds I'm trying.

Here are the inspection properties:

[{
    "Args": [
        "-c",
        "#(nop) ADD file:e603718edfc97b931820b7d37435f19463fdaa5469c0605d253380c90c9ac901 in /app/init"
    ],
    "Config": {
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": null,
        "CpuShares": 0,
        "Cpuset": "",
        "Domainname": "",
        "Entrypoint": null,
        "Env": [
            "HOME=/",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "DEBIAN_FRONTEND=noninteractive"
        ],

        "Hostname": "0e6756a12879",
        "Image": "9f6d25fddff7374e3ebf541351dabd41650a52d0284ee51060dc0b81dd557631",
        "Memory": 0,
        "MemorySwap": 0,
        "NetworkDisabled": false,
        "OnBuild": [],
        "OpenStdin": false,
        "PortSpecs": null,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": null,
        "WorkingDir": ""
    },
    "Created": "2014-07-01T20:37:21.353676702Z",
    "Driver": "devicemapper",
    "ExecDriver": "native-0.2",
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "Dns": null,
        "DnsSearch": null,
        "Links": null,
        "LxcConf": null,
        "NetworkMode": "",
        "PortBindings": null,
        "Privileged": false,
        "PublishAllPorts": false,
        "VolumesFrom": null
    },
    "HostnamePath": "",
    "HostsPath": "",
    "Id": "9bcc6cd06c833f8f0275c48bf7de6b15e8f73a8a799f43056d2ff95b91a4fa76",
    "Image": "9f6d25fddff7374e3ebf541351dabd41650a52d0284ee51060dc0b81dd557631",
    "MountLabel": "",
    "Name": "/naughty_nobel",
    "NetworkSettings": {
        "Bridge": "",
        "Gateway": "",
        "IPAddress": "",
        "IPPrefixLen": 0,
        "PortMapping": null,
        "Ports": null
    },
    "Path": "/bin/sh",
    "ProcessLabel": "",
    "ResolvConfPath": "",
    "State": {
        "ExitCode": 0,
        "FinishedAt": "0001-01-01T00:00:00Z",
        "Paused": false,
        "Pid": 0,
        "Running": false,
        "StartedAt": "0001-01-01T00:00:00Z"
    },
    "Volumes": null,
    "VolumesRW": null

@michaelbarton

I am also getting the 'invalid argument' error listed above. I am running docker on an Amazon m2.large instance running Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64).

Here is the output of docker info

Containers: 10
Images: 72
Storage Driver: devicemapper
 Pool Name: docker-202:16-475138-pool
 Data file: /mnt/docker/devicemapper/devicemapper/data
 Metadata file: /mnt/docker/devicemapper/devicemapper/metadata
 Data Space Used: 2574.9 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 4.5 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 3.13.0-24-generic
WARNING: No swap limit support

Here is the output of docker version:

Client version: 1.0.1
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 990021a
Server version: 1.0.1
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 990021a

I am running docker as follows:

docker run \
   --volume /mnt/dir_1:/outputs:rw \
   --volume /mnt/dir_2:/inputs:ro \
   --detach=false \
   --cidfile=/mnt/dir_3/container_id \
   my_image \
   arg_1 \
   arg_2

However I get the following error:

2014/07/01 22:26:59 Error response from daemon: Error mounting '/dev/mapper/docker-202:16-475138-e1d927e16afbf1537f66bb0b6f3df56be7160cea7abefe19059aff2c00c55a0a-init' on '/mnt/docker/devicemapper/mnt/e1d927e16afbf1537f66bb0b6f3df56be7160cea7abefe19059aff2c00c55a0a-init': invalid argument

I already ran this command successfully once and it produced the expected output. The second time it produces this error.

@rthomas
Contributor
rthomas commented Jul 3, 2014

We have also been seeing this - at first we thought it was the libcontainer backend and switched back to lxc but we have see the issue resurface.

We have also seen some correlation around the oom killing of processes and this issue come up, though that could be a red herring.

Again correlation, but we have also seen this occur when we attempt to start a number of containers in quick succession.

Once the devicemapper device becomes corrupted like this, we are unable to start any more containers on that instance - we either reprovision it, or stop docker, frag the directory and start again.

This is on AWS on an m3.large instance.

After the corruption occurred I started docker in debug mode - the output can be seen here: http://pastie.org/pastes/9351622/text

As another note, it is only parts of the mount that become corrupted - we've been able to detect them by running something that would traverse all files (e.g. find or du -sh /mnt/docker/devicemapper). Running an e2fsck on the devices in question confirms this however the recovered data is useless (filled up lost+found).

OOM Killer syslog

Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000657] BUG: Bad page map in process java  pte:8000000000000325 pmd:1d11ab067
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000670] addr:00007f13f9097000 vm_flags:08000071 anon_vma:          (null) mapping:          (null) index:7f13f9097
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000676] CPU: 0 PID: 19624 Comm: java Tainted: GF   B      O 3.13.0-24-generic #47-Ubuntu
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000679]  ffff8800f5feaa80 ffff88014efefa98 ffffffff81715ac4 00007f13f9097000
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000682]  ffff88014efefae0 ffffffff81174183 80000002419cb325 00000007f13f9097
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000684]  ffff8801d11ab4b8 80000002419cb325 00007f13f9097000 00007f13f9098000
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000686] Call Trace:
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000693]  [<ffffffff81715ac4>] dump_stack+0x45/0x56
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000697]  [<ffffffff81174183>] print_bad_pte+0x1a3/0x250
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000699]  [<ffffffff81175439>] vm_normal_page+0x69/0x80
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000701]  [<ffffffff8117580b>] unmap_page_range+0x3bb/0x7f0
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000703]  [<ffffffff81175cc1>] unmap_single_vma+0x81/0xf0
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000706]  [<ffffffff81176d39>] unmap_vmas+0x49/0x90
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000708]  [<ffffffff8117feec>] exit_mmap+0x9c/0x170
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000712]  [<ffffffff8110fcf3>] ? __delayacct_add_tsk+0x153/0x170
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000715]  [<ffffffff8106482c>] mmput+0x5c/0x120
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000718]  [<ffffffff81069bbc>] do_exit+0x26c/0xa50
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000721]  [<ffffffff8108e4aa>] ? hrtimer_cancel+0x1a/0x30
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000724]  [<ffffffff810d8442>] ? futex_wait+0x1b2/0x290
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000726]  [<ffffffff8106a41f>] do_group_exit+0x3f/0xa0
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000730]  [<ffffffff8107a050>] get_signal_to_deliver+0x1d0/0x6f0
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000734]  [<ffffffff81013448>] do_signal+0x48/0x960
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000736]  [<ffffffff8101b7d9>] ? sched_clock+0x9/0x10
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000739]  [<ffffffff8109d13d>] ? sched_clock_local+0x1d/0x80
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000741]  [<ffffffff811112fc>] ? acct_account_cputime+0x1c/0x20
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000743]  [<ffffffff8109d76b>] ? account_user_time+0x8b/0xa0
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000745]  [<ffffffff8109dd84>] ? vtime_account_user+0x54/0x60
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000747]  [<ffffffff81013dc9>] do_notify_resume+0x69/0xb0
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.000750]  [<ffffffff8172676a>] int_signal+0x12/0x17
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.060404] docker0: port 3(vethX3PALP) entered disabled state
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.060969] device vethX3PALP left promiscuous mode
Jul  2 23:18:45 mesos-slave-3 kernel: [1212008.060977] docker0: port 3(vethX3PALP) entered disabled state

root@mesos-slave-3:/var/log# docker info
Containers: 4
Images: 181
Storage Driver: devicemapper
 Pool Name: docker-202:16-1785858-pool
 Data file: /mnt/docker/devicemapper/devicemapper/data
 Metadata file: /mnt/docker/devicemapper/devicemapper/metadata
 Data Space Used: 6321.8 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 16.8 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: lxc-1.0.3
Kernel Version: 3.13.0-24-generic
Debug mode (server): true
Debug mode (client): false
Fds: 14
Goroutines: 15
EventsListeners: 0
Init Path: /usr/bin/docker
WARNING: No swap limit support
root@mesos-slave-3:/var/log# docker version
Client version: 1.0.0
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 63fe64c
Server version: 1.0.0
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 63fe64c
@michaelbarton

@rthomas When you say that you tried switching to lxc is that bare lxc-containers or different backend for docker? I have thought about trying different storage options other than devmapper, such as aufs. I believe this can be set with an option to the docker daemon. If the mount is corrupted I am not sure how much this could help though.

@michaelbarton

I should also note that I am also running a lot of containers for one-off commands and then removing them afterwards as each one uses a lot of disk space. So this perhaps my issue is also something related to corruption.

@rthomas
Contributor
rthomas commented Jul 3, 2014

@michaelbarton no we changed to using lxc as the execution backend for docker.

We don't really care about the mount at the moment, we just don't want this happening again and again as it requires us to manually intervene and clean things up.

We haven't tried aufs as I believe a previous comment mentioned it was removed in 1.0.

@rthomas
Contributor
rthomas commented Jul 4, 2014

I've tried with both the busybox loop example in the first comment, and with this command: for i in {0..100}; do docker run --rm -m 512k java java & done (to test the container being killed) both with other containers running and with nothing else running and I can't reproduce this on the same machine where I had previously seen it.

Another observation from the previous time I have seen it, is that the corruption seems to occur after a while - which may explain why I cannot reproduce it right now. We run 4 slaves in two environments at the moment and over a long enough period of time this occurs on all instances (about a week or two of usage).

@rthomas
Contributor
rthomas commented Jul 6, 2014

This may be more correlation, but on every one of the devices that is corrupted the /rootfs/proc directory is corrupted and from what I can see this is where the first error occurs (according to stats in debugfs).

root@mesos-slave-3:/mnt/docker.old/devicemapper/mnt# debugfs
debugfs 1.42.9 (4-Feb-2014)
debugfs:  open /dev/mapper/docker-202:16-1785858-eb454d1d2ccff8075ad0e3b4b766d1a343292babbc2f17f99f164e990db8663e
debugfs:  ls /rootfs/proc
/rootfs/proc: EXT2 directory corrupted
debugfs:

I'd love to try and find a solution for this - as an experiment I have switched one of our nodes over to xfs.

The other bit of info I mentioned in an earlier comment is the timing - it appears to happen after about 14 days of uptime - three of our slaves had this occur within ~1 day of each other all with 14 days of uptime.

@rthomas
Contributor
rthomas commented Jul 8, 2014

This has occurred again, when provisioning multiple containers. We see these errors in the syslogs after the corruption occurs.

[1655388.459054] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: discard
[1655388.471664] EXT4-fs error (device dm-3): ext4_free_inode:323: comm docker: bit already cleared for inode 525284
[1655388.473000] EXT4-fs error (device dm-3): ext4_lookup:1437: inode #524289: comm docker: deleted inode referenced: 524386
[1655388.473073] EXT4-fs error (device dm-3): ext4_lookup:1437: inode #524289: comm docker: deleted inode referenced: 524386
@rthomas
Contributor
rthomas commented Jul 8, 2014

For containers that fail to start with the input/output error we see this in the docker logs

[debug] deviceset.go:252 registerDevice(52, a2cda4f6409ef5a4c9b4ba9c8a93e61b9664b7fa7c84e02eb7bbadbcc8995515-init)
[debug] deviceset.go:278 activateDeviceIfNeeded(a2cda4f6409ef5a4c9b4ba9c8a93e61b9664b7fa7c84e02eb7bbadbcc8995515-init)
[debug] deviceset.go:992 [devmapper] UnmountDevice(hash=a2cda4f6409ef5a4c9b4ba9c8a93e61b9664b7fa7c84e02eb7bbadbcc8995515-init)

When a container starts though, we actually see the non-init container get registered and started before unmounting the init.

[debug] deviceset.go:252 registerDevice(53, 8d619dfabd6f8edc1709f87b5ca559250c4dee77d73f5b1ea531b5de05b066e5-init)
[debug] deviceset.go:278 activateDeviceIfNeeded(8d619dfabd6f8edc1709f87b5ca559250c4dee77d73f5b1ea531b5de05b066e5-init)
[debug] deviceset.go:252 registerDevice(54, 8d619dfabd6f8edc1709f87b5ca559250c4dee77d73f5b1ea531b5de05b066e5)
[debug] deviceset.go:992 [devmapper] UnmountDevice(hash=8d619dfabd6f8edc1709f87b5ca559250c4dee77d73f5b1ea531b5de05b066e5-init)
@rthomas
Contributor
rthomas commented Jul 8, 2014

This also occurred on our slave that was using xfs on devicemapper - but with a different error message.

Error: stat /mnt/docker/devicemapper/mnt/dee2458d04cf7d1d367ae609892495c1616f90ee45d08c176015ddd55da3ed07-init/rootfs/dev/pts: structure needs cleaning
@son1cc
son1cc commented Jul 9, 2014

Happened for me on Ubuntu 14.04 with Docker 1.0.1

Removing intermediate container 766f06d8d7c2
Step 4 : COPY http://varnishStart.sh /usr/local/bin/run
Service 'varnish' failed to build: Error getting container init rootfs 186d0e902fd4d1b8394dbb274ff31e4c31b681bbf02253a4fafffb51b23231b2 from driver devicemapper: Error mounting '/dev/mapper/docker-202:96-5505025-186d0e902fd4d1b8394dbb274ff31e4c31b681bbf02253a4fafffb51b23231b2-init' on '/var/lib/docker/devicemapper/mnt/186d0e902fd4d1b8394dbb274ff31e4c31b681bbf02253a4fafffb51b23231b2-init': no such file or directory
Build step 'Execute shell' marked build as failure
Finished: FAILURE

@rthomas
Contributor
rthomas commented Jul 9, 2014

We have switched away from devicemapper and are using the btrfs backend now, we have yet to see this issue repeat itself across 8 slaves.

@son1cc
son1cc commented Jul 9, 2014

is the newest version of the code using btrfs?

@rthomas
Contributor
rthomas commented Jul 9, 2014

You can change to the btrfs storage backend using the -s btrfs option for the daemon, you do however need the btrfs-tools package installed and a btrfs formatted mount that docker is pointing at for it to work.

@michaelbarton

@rthomas Thanks for posting your solution to this problem. You mentioned above that you're running on EC2. Would you be able to share a code snippet or point to an example of using btrfs on EC2? I've tried googling but I can't find an example of how to mount a drive formatted this way. I would like to try this to see if it fixes the problems I encountered.

@rthomas
Contributor
rthomas commented Jul 10, 2014

@michaelbarton yes, this is on EC2 instances with the ephemeral volume attached as the docker target and running Ubuntu 14.04.

I did the following, this will destroy all data on the ephemeral volume!

sudo -i
apt-get install btrfs-tools
umount /mnt
# This assumes you have the normal ubuntu setup with /dev/xvdb mounted as /mnt and registered in your /etc/fstab
mkfs.btrfs -f /dev/xvdb
mount /mnt

This will give you a btrfs volume on /mnt, you now need to change docker to use the btrfs storage backend and to use the /mnt/docker directory. Edit the /etc/default/docker file and set the DOCKER_OPTS to have the -s btrfs and -g /mnt/docker options.

I found there were issues unmounting the /mnt target when there were devicemapper mounts in there, if you haven't been using the ephemeral volume though you shouldn't have any issues.

Hope it helps you!

@michaelbarton

@rthomas Thanks for sharing this code. This worked for me too - using btrfs solved this error and I no longer had any problems.

@unclejack unclejack removed their assignment Jul 24, 2014
@hackerfriendly

@rthomas Thank you. That seems to have done the trick for me too.

For search engine posterity, my specific error was:

2014/07/31 17:38:17 Unknown filesystem type on /dev/mapper/docker-202:16-41877506-577b700d00b4f915903c4fef4dafa29116fbb3e9cb5d0ea197ac6aefb67a3ad8-init

...during a 'docker build' on docker 1.0.0 running on an Ubuntu 14.04 AWS instance. Changing from devicemapper to btrfs solved the immediate issue for me.

@jaredm4
jaredm4 commented Aug 12, 2014

Also happened to me. Docker client 1.0.0, daemon version 1.1.2. If I rerun the exact same steps to reproduce, it usually doesn't fail again.

Step 8 : ADD files/supervisor-job-queue.conf /etc/supervisor/conf.d/
2014/08/12 22:02:15 Error getting container init rootfs 7307882f55620eefb7716d598fbc107e435c141837313107561bb6bf088ea418 from driver devicemapper: Error mounting '/dev/mapper/docker-202:1-397765-7307882f55620eefb7716d598fbc107e435c141837313107561bb6bf088ea418-init' on '/var/lib/docker/devicemapper/mnt/7307882f55620eefb7716d598fbc107e435c141837313107561bb6bf088ea418-init': no such file or directory

This is running on Ubuntu 14.04 on an EC2 large with an SSD EBS volume attached (however the folder it is trying to mount is not part of the EBS).

@openfirmware

I am running into this issue while trying to use Dokku to deploy some Rails apps. My host is an OpenStack instance running Ubuntu Server Trusty 14.04.1 LTS, with kernel 3.13.0-34-generic. I was using devicemapper as the storage backend, and I am up to the point almost every build or run command triggers the error response from daemon: Error getting container CONTAINER_ID devicemapper issue. This is on Docker 1.2.0.

So I removed Docker and set up AUFS (it still works in 1.2.0 — see below). Here is a quick uninstall/reinstall; note that this deletes all images and containers!

$ sudo -i
# service docker stop
# apt-get remove lxc-docker
# apt-get autoremove
# rm -rf /var/lib/docker
# apt-get update
# apt-get install linux-image-extra-`uname -r`
# apt-get install lxc-docker
# docker info
Containers: 9
Images: 21
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 39
Execution Driver: native-0.2
Kernel Version: 3.13.0-34-generic
Operating System: Ubuntu 14.04.1 LTS
WARNING: No swap limit support

If you install the linux-image-extra package before Docker, it will enable AUFS support. This is mentioned on the Ubuntu install docs, but it is noted below the 13.04/13.10 instructions.

I just made the switchover to AUFS, and so far Dokku is rebuilding my apps without issue. Deploys are also way faster, probably by an order of magnitude.

@hmalphettes

Thanks @openfirmware: on an existing ubuntu VM installing aufs and reinstalling docker was the easiest.

@rthomas cheers! That hints at why coreos provides docker is configured with btrfs and I would only get those issues with ubuntu (14.04, docker-1.2).

@soichih
soichih commented Sep 5, 2014

I am having a similar problem with Docker 1.1.2 on RHEL 6.5.

If I have running containers and do "service docker start", none of the container will start up again. When I try to manually start them up, I get ..

docker start influxdb
Error response from daemon: Cannot start container influxdb: Error getting container     34533992c8102c47e5f0d637f7cd38b1ed989a2e5ecaeed7c7eb66fac7731a07 from driver devicemapper:     Error mounting '/dev/mapper/docker-252:17-3457090-  34533992c8102c47e5f0d637f7cd38b1ed989a2e5ecaeed7c7eb66fac7731a07' on   '/usr/local/docker/devicemapper/mnt/34533992c8102c47e5f0d637f7cd38b1ed989a2e5ecaeed7c7eb66fac7731a07': device or resource busy
2014/09/05 21:50:24 Error: failed to start one or more containers

I have to umount manually on each container, then I can start them up again.

@ndevenish

I'm having this problem on 1.2.0 on ubuntu 14.04. It's a brand new high end i7, running Xen Server and on an SSD - ubuntu runs blindingly fast, so if it's a race condition this is probably exacerbating it.

It's so bad that larger docker build scripts (10+ RUNs) almost always hit the problem and need to be re-run.

@peterloron

Also seeing this in ubuntu 14.04 + docker 1.2.0.

@abrkn
abrkn commented Sep 17, 2014

+1 ubuntu 14:04 + docker 1.2.0

@rafecolton
Contributor

I've also been having a slew of issues with devicemapper on ubuntu 14.04 w/ docker 1.2.0. FWIW, I switched back to the aufs graph storage a couple weeks ago and haven't looked back since

@smerrill
Contributor

@soichih, Could I enlist you to try the change in #8094 to the initscript? The initscript will kill -9 the docker daemon if it doesn't shut down cleanly in 3 seconds (killproc's default.) I find that many applications in containers (MySQL, Sphinx, etc.) take longer than 3 seconds to respond to a stop and as a result, they will not come back up cleanly when the docker daemon restarts via the initscript.

@soichih
soichih commented Sep 18, 2014

I've recently updated the kernel / glibc / initscripts, etc.. and when I rebooted it, my docker container came back up as expected. I am not sure if something got fixed, or my container just happen to restart really quickly.. but I can't seem to recreate this problem anymore.

I did try adding "-d 300" on my docker init script and restarted it, and all container came back as expected

(update...)

Scratch that.. I just installed docker on a new CentOS 6 VM and I am having this problem again. This is a very easily reproducible, and a rather serious bug. I don't know why it's taking so long for Docker developers to fix this.. (I am guessing it's not even discussed..?)

@jmreicha

Exact same issue, Docker 1.2.0 and Ubuntu 14.04 with 3.13 kernel.

The method @openfirmware describes worked for me.

@sniperd
sniperd commented Sep 22, 2014

Same issue with Docker 1.2.0 + Ubuntu 14.04 3.13.0 kernel.

@vbatts
Contributor
vbatts commented Sep 23, 2014

@unclejack this issue looks to have become a catch-all for similar issues.

I've just tested with:

DOCKER=${DOCKER:-docker}

for i in {1..50}; do
        CID=`${DOCKER} run -d -P registry:latest`
        for j in {1..20}; do
                ${DOCKER} kill $CID 2> /dev/null
                ${DOCKER} restart $CID
        done
        ${DOCKER} kill $CID 2> /dev/null
        ${DOCKER} rm $CID 2> /dev/null &
done

on the ubuntu

Kernel Version: 3.13.0-36-generic
Operating System: Ubuntu 14.04.1 LTS

Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.2.1
Git commit (client): fa7b24f
OS/Arch (client): linux/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.2.1
Git commit (server): fa7b24f

and fedora

Kernel Version: 3.16.2-201.fc20.x86_64
Operating System: Fedora 20 (Heisenbug)

Client version: 1.2.0-dev
Client API version: 1.15
Go version (client): go1.2.2
Git commit (client): 9941b03
OS/Arch (client): linux/amd64
Server version: 1.2.0-dev
Server API version: 1.15
Go version (server): go1.2.2
Git commit (server): 9941b03

So, 1.2.0 and HEAD, both on devicemapper, and I am not seeing the issue you originally reported here. It seems there are a couple of issues that folks are reporting here.

@Neetuj
Neetuj commented Sep 24, 2014

facing the same issue

(buzzenv)njain@njain3:~/office/buzz$ sudo docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.1
Git commit (client): fa7b24f
OS/Arch (client): linux/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.3.1
Git commit (server): fa7b24f

(buzzenv)njain@njain3:~/office/buzz$ uname -a
Linux njain3 3.2.0-60-virtual #91-Ubuntu SMP Wed Feb 19 04:13:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

(buzzenv)njain@njain3:~/office/buzz$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.4 LTS
Release: 12.04
Codename: precise

and i am getting this :-
(buzzenv)njain@njain3:~/office/buzz$ sudo docker run busybox /bin/echo hello world
Unable to find image 'busybox' locally
Pulling repository busybox
a9eb17255234: Error pulling image (latest) from busybox, Driver devicemapper failed to create image rootfs 42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229: device 42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229 already exists 90234240388d6297bc0f32c34229 already exists
42eed7f1bf2a: Error downloading dependent layers
2014/09/24 14:42:35 Error pulling image (latest) from busybox, Driver devicemapper failed to create image rootfs 42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229: device 42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229 already exists

any pointers ?

@stephanbuys

@mbentley I have to agree with @memelet, the only clue that AUFS is required is in the raw source of the install script at https://get.docker.com, the install page at https://docs.docker.com/installation/ubuntulinux/ only mentions AUFS in the context of older versions. This should either be handled by docker or at least mentioned as a prerequisite. My gut feel on this is that devicemapper should actually "just work" even if it is sub-optimal, or it should be disabled until enabled.

@IcaroBichir

I'm having the same issue reported.

time="2015-06-10T17:24:26-03:00" level=fatal msg="Error response from daemon: 
Cannot start container 16e1b3fabf3c9d9c9383f9ab59fef558417bb61020dbbb19b947540af6999b78: 
Error getting container 16e1b3fabf3c9d9c9383f9ab59fef558417bb61020dbbb19b947540af6999b78 from driver devicemapper: 
Error mounting '/dev/mapper/docker-202:1-531040-16e1b3fabf3c9d9c9383f9ab59fef558417bb61020dbbb19b947540af6999b78' on '/var/lib/docker/devicemapper/mnt/16e1b3fabf3c9d9c9383f9ab59fef558417bb61020dbbb19b947540af6999b78': no such file or directory" 
$ docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64
$ docker info
Containers: 0
Images: 67
Storage Driver: devicemapper
 Pool Name: docker-202:1-531040-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.321 GB
 Data Space Total: 107.4 GB
 Data Space Available: 66.12 GB
 Metadata Space Used: 2.925 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.145 GB
 Udev Sync Supported: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 3.13.0-49-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 3.859 GiB
@ashish235

Any progress on this? I 'm still getting this issue, a reboot also didn't help.

I 've a CentOS 7.0.

[root@localhost ansible_poc]# docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

INFO[0015] Error getting container 3f8d6305bc3fd33f67a8fbcc06c0d08a6bcee116e9330c08a3f5eaa1721a7878 from driver devicemapper: open /dev/mapper/docker-253:1-13461186-3f8d6305bc3fd33f67a8fbcc06c0d08a6bcee116e9330c08a3f5eaa1721a7878: no such file or directory

@vbatts
Contributor
vbatts commented Jun 11, 2015

@ashish235 docker info as well, please

@vbatts
Contributor
vbatts commented Jun 11, 2015

This issue is resolved

Key item to observe before commenting on this issue, is whether docker info | grep Udev returns for Udev Sync Supported: true or not.

Udev sync

Devicemapper storage driver expects to be synchronized with udev. When this info reports false, that means that devicemapper and udev are not able to sync. In this event there is a race condition that many of you have experienced.

Causes

There are a couple of causes for Udev sync to not be supported:

  • The docker binary used on your host, was statically linked when compiled, instead of dynamically linking.
  • If your docker binary is dynamically linked, the version of libdevmapper.so too old to support Udev sync

Solutions

  1. Install the docker binary from your distribution (i.e. the docker rpm from fedora is dynamically linked)
  2. Build the docker binary yourself, dynamically linked
    • git clone git://github.com/docker/docker.git
    • cd docker
    • AUTO_GOPATH=1 ./hack/make.sh dynbinary
  3. If your docker binary is dynamic (test with file $(which docker) | grep dynamically) and Udev sync reports false, you will likely need to update the distribution release used

Furthermore, in docker 1.7 this will become extremely clear when the daemon will error out when starting, if Udev sync is not supported. Familiarize yourself with getting a dynamic linked binary and/or updating your distribution release before this.

@IPvFletch

Anyone know when the Yum repository will have the dynamically linked binary for 1.7? I have tried compiling it and it keeps failing. Even updated to golang 1.4.

@mbentley
Contributor

There are currently RPMs available for the 1.7 RC so I would assume so: #13528

@memelet
memelet commented Jun 12, 2015

For Ubuntu at least, switching to AUFS has resolved the problem. But are
there any disadvantages to AUFS over devicemapper?
On Jun 11, 2015 9:59 PM, "Matt Bentley" notifications@github.com wrote:

There are currently RPMs available for the 1.7 RC so I would assume so:
#13528 #13528


Reply to this email directly or view it on GitHub
#4036 (comment).

@ashish235

@vbatts, below is my dokcer info and Udev sync status.

[root@localhost ~]# docker info
Containers: 17
Images: 263
Storage Driver: devicemapper
Pool Name: docker-253:1-13461186-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 22.92 GB
Data Space Total: 107.4 GB
Data Space Available: 14.88 GB
Metadata Space Used: 21.6 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.126 GB
Udev Sync Supported: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 15.4 GiB
Name: localhost.localdomain
ID: EUNO:GZDS:6NCD:5XUN:OLCB:XCRO:QN6Q:YJ44:BASK:X3T7:T33Q:6RUY

[root@localhost ~]# docker info | grep Udev
Udev Sync Supported: false

I will try out your solution and see if that works for CentOS and Oralce Linux. The problem is the issue can't be reproduced at will or maybe I don't know how to reproduce.

Thanks, Cheers!

@IcaroBichir

Guys, I've made a tutorial, how to rebuild and resolve the problem between Docker v-1.6.2 and Udev.
If you have further questions, ask on my site!

http://icarobichir.com.br/posts/building-a-dynamically-linked-docker-v-1-6-2/

See ya

@dachary
dachary commented Jun 14, 2015

@IcaroBichir URL please ?

@dcosson
dcosson commented Jun 16, 2015

So as far as I can tell the official docker.com instructions still recommend installing via the https://get.docker.com/ script, on ubuntu that installs from the apt source https://get.docker.com/ubuntu docker main. For me installing the latest lxc-docker from here I get docker 1.6.2 and it shows Udev sync supported: false.

Just want to clarify that this is expected behavior since it's not mentioned above. Will docker 1.7 fix this issue in the official apt repo?

@mbentley
Contributor

Using the get.docker.com script will also default to AUFS instead of devicemapper.

@dcosson
dcosson commented Jun 16, 2015

@mbentley ahh, should have read further up the thread. Confirmed that this seems to work for me, thanks.

@AlbanMontaigu AlbanMontaigu added a commit to AlbanMontaigu/boot2docker-vagrant-template that referenced this issue Jun 25, 2015
@AlbanMontaigu AlbanMontaigu Come back to aufs by default because of docker 1.7 19c5b4b
@sitamet
sitamet commented Jun 26, 2015

I still have errors

Handler for POST /containers/{name:.*}/start returned error: Cannot start container cd67a85...: Error getting container cd67a854b... from driver devicemapper: Error mounting '/dev/devicemapper/docker-9:2-4982238-cd67


HTTP Error: statusCode=404 Cannot start container cd67a854bd3189....: Error getting container cd67a854bd31891....08c3 from driver devicemapper: Error mounting '/dev/mapper/docker-9:2-4982238

And I am running:

CentOS Linux release 7.1.1503
3.14.32-xxxx-grs-ipv6-64
$ docker info
Containers: 140
Images: 164
Storage Driver: devicemapper
 Pool Name: docker-9:2-4982238-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/direct-lvm/data
 Metadata file: /dev/direct-lvm/metadata
 Data Space Used: 4.022 GB
 Data Space Total: 102 GB
 Data Space Available: 97.98 GB
 Metadata Space Used: 19.63 MB
 Metadata Space Total: 5.369 GB
 Metadata Space Available: 5.349 GB
 Udev Sync Supported: true
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.14.32-xxxx-grs-ipv6-64
Operating System: CentOS Linux 7 (Core)
CPUs: 8
Total Memory: 62.94 GiB

it is a dynamic build:

# file $(which docker) | grep dynamically
/usr/bin/docker: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0xd9b3ec9d81dc29fe4095d49fab0479895434835e, not stripped

And the rpm I use are:

docker.x86_64  1.6.2-14.el7.centos  
device-mapper-devel.i686               7:1.02.93-3.el7_1.1                                                                                                                     updates  
device-mapper-devel.x86_64             7:1.02.93-3.el7_1.1                                                                                                                     updates  
device-mapper-event-devel.i686         7:1.02.93-3.el7_1.1                                                                                                                     updates  
device-mapper-event-devel.x86_64       7:1.02.93-3.el7_1.1                                                                                                                     updates  
device-mapper-event-libs.i686          7:1.02.93-3.el7_1.1                                                                                                                     updates  
device-mapper-libs.i686                7:1.02.93-3.el7_1.1                                                                                                                     updates  

What am I doing wrong :(

@garrickp

Opinion: This is sub-optimal resolution to this problem for three reasons:

  • Ubuntu 14.04 (not to mention CentOS and RHEL) is a fairly common OS to see in production environments
  • AUFS' performance is measurably worse than devicemapper, and support for AUFS by the Ubuntu team has been dropped due to lack of it making upstream. This indicates that AUFS installs on Ubuntu are effectively unsupported.
  • The Docker provided install scripts perform some OS and version specific magic during the install, but still require their users to manually build Docker if they want to use a file system still supported by the vendor.

It might be considered acceptable to put the burden of a workaround on the user if this were Slackware or Gentoo with their comparatively marginal install bases, but for Ubuntu, CentOS and RHEL, it's silly.

This could be fixed by the Docker team in the package build process, but hasn't. Why foist an outdated and unsupported storage engine on Docker's users, and call it a solution?

@RoelVdP
RoelVdP commented Jul 1, 2015

I too believe this issue should be re-opened & re-evaluated.

@Offirmo
Offirmo commented Jul 1, 2015

I would add that Ubuntu is the official target of docker...

@rhvgoyal
Contributor
rhvgoyal commented Jul 1, 2015

@sitamet

Have you done any storage configuration changes? Like changing from loop devices to block devices or anything else? Often we have the case that people change storage configuration and docker metadata is not valid for new configuration.

@rhvgoyal
Contributor
rhvgoyal commented Jul 1, 2015

@sitamet

Are you able to reproduce the issue consistently? How do I go about reproducing it?

Is it happening with only that particular container or rest of the containers too.

@sitamet
sitamet commented Jul 1, 2015

It happens ONLY after running aprox 100-120 containers. (I have plenty of
disk space)

Then when the problem starts, its persistent and almost always present on
every new container run.

El dimecres, 1 de juliol de 2015, Vivek Goyal notifications@github.com va
escriure:

@sitamet https://github.com/sitamet

Are you able to reproduce the issue consistently? How do I go about
reproducing it?

Is it happening with only that particular container or rest of the
containers too.


Reply to this email directly or view it on GitHub
#4036 (comment).

@selipso
selipso commented Jul 15, 2015

I am still getting this issue when building the MeanJS Dockerfile using Ubuntu 14.04. Only changes I've made are changing the first line in the Dockerfile to FROM google/nodejs and renaming fig.yml to docker-compose.yml

After running docker-compose build, I get the following error:

Step 10 : ADD . /home/mean
Service 'web' failed to build: Error getting container 2f644f7a1c63b91f525603988e5b418a8063c3575162c0755eb8f0f90ce28fa7 from driver devicemapper: Error mounting '/dev/mapper/docker-252:0-3932389-2f644f7a1c63b91f525603988e5b418a8063c3575162c0755eb8f0f90ce28fa7' on '/var/lib/docker/devicemapper/mnt/2f644f7a1c63b91f525603988e5b418a8063c3575162c0755eb8f0f90ce28fa7': no such file or directory

I am using

Containers: 7
Images: 186
Storage Driver: devicemapper
 Pool Name: docker-252:0-3932389-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 6.096 GB
 Data Space Total: 107.4 GB
 Data Space Available: 101.3 GB
 Metadata Space Used: 11.6 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.136 GB
 Udev Sync Supported: false
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-43-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 8
Total Memory: 15.67 GiB

Edit: I fixed the issue by finding containers using docker ps -a and removing containers that were not running

@clzhao
clzhao commented Jul 16, 2015

@hqhq @vbatts
After I cloned the lvm2, and tried to run "./configure --enable-udev_sync" . Errors occurred as follows:

checking for PKGCONFIGINIT... no
pkg-config initialized
checking for UDEV... no
configure: error: Package requirements (libudev >= 143) were not met:

No package 'libudev' found

Then I tried to install "libudev-147-2.57.el6.x86_64.rpm" with the command rpm -ivh --replacefiles libudev-147-2.57.el6.x86_64.rpm The result is

[root@c208 /usr/local/eric_tmp]# rpm -ivh --replacefiles libudev-147-2.57.el6.x86_64.rpm 
Preparing...                ########################################### [100%]
    package libudev-147-2.57.el6.x86_64 is already installed

Then I went back to /usr/local/lvm2 . But when I tried to run ./configure --enable-udev_sync. The result was still as previous

checking for UDEV... no
configure: error: Package requirements (libudev >= 143) were not met:

No package 'libudev' found

My system is Centos6.

So How to fix the problem? Thank you very much.

@rhvgoyal
Contributor

static libudev is not built or shipped. It is not supported by systemd project.

@jonatanblue

@selipso you have "Storage Driver: devicemapper" and "Udev Sync Supported: false" and so you are having a bad time, as explained by @vbatts #4036 (comment). See also #10705.

@selipso
selipso commented Jul 21, 2015

I see, ok thanks

On Jul 21, 2015, at 6:32 AM, jonatanblue notifications@github.com wrote:

@selipso you have "Storage Driver: devicemapper" and "Udev Sync Supported: false" and so you are having a bad time, as previous comments have pointed out. See also #10705.


Reply to this email directly or view it on GitHub.

@jonatanblue

@selipso The new dynamic binaries should allow udev sync to work properly
http://blog.docker.com/2015/07/new-apt-and-yum-repos/

@moolen
moolen commented Jul 29, 2015

@jonatanblue

HOLY SHIT I HAVE TO CAPSLOCK.

The new dynamic binaries work for me on debian wheezy with udev support enabled (YEY!)

@vbatts
Contributor
vbatts commented Jul 29, 2015

yes. :-)

On Wed, Jul 29, 2015 at 3:27 PM, meeeh notifications@github.com wrote:

@jonatanblue https://github.com/jonatanblue

HOLY SHIT I HAVE TO CAPSLOCK.

The new dynamic binaries work for me on debian wheezy with udev support
enabled (YEY!)


Reply to this email directly or view it on GitHub
#4036 (comment).

@billbok
billbok commented Aug 3, 2015

Hello,

I have the same error when i build dockerfile.

Error getting container eb8c102fb61b95984efaac7a160172e6996f3d64ee563272d4e0386d48f824cc from driver devicemapper: open /dev/mapper/docker-8:3-91488259-eb8c102fb61b95984efaac7a160172e6996f3d64ee563272d4e0386d48f824cc: no such file or directory
(with docker.io 1.6)

So, i follow this procedure:
http://blog.docker.com/2015/07/new-apt-and-yum-repos/
... but I've the same error. I've test with another Dockerfile=> same error

I run on debian 8.1

docker -v
Docker version 1.7.1, build 786b29d

docker info
docker info
Containers: 15
Images: 328
Storage Driver: devicemapper
Pool Name: docker-8:3-91488259-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 13.48 GB
Data Space Total: 107.4 GB
Data Space Available: 93.9 GB
Metadata Space Used: 18.14 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.129 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /data/docker/devicemapper/devicemapper/data
Metadata loop file: /data/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.90 (2014-09-01)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.14.32-xxxx-grs-ipv6-64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 4
Total Memory: 15.57 GiB
Name: ns3008012.ip-151-80-44.eu
ID: G7DE:NM7H:R4SZ:W365:VL3Y:KE6L:UA33:RRYO:QALJ:6WH4:3RPH:I6KZ

Any idea?

thank you...
Sorry for my poor english

@lukebayes

For the record, we're running Ubuntu trusty (14.04) on AWS EC2 and ran into this issue today on Docker 1.7.1.

The solution above that states, "This issue is resolved" did not work. When attempting to build from the v1.7.1 branch, the build failed with:

/home/ubuntu/src/docker/pkg/devicemapper/devmapper_wrapper.go:7:26: fatal error: libdevmapper.h: No such file or directory
  • Restarting the virtual machine did not solve the problem
  • Removing all local images (and then restarting) did not solve the problem

Doing the following fixed the issue for us:
https://blog.docker.com/2015/07/new-apt-and-yum-repos/

@mbentley
Contributor
mbentley commented Aug 4, 2015

@lukebayes FYI, that just means that you were missing a package in order to be able to build the binary; using the new apt repos fixes the issue since the binaries are dynamic (which is the end goal of building yourself anyway)

@happyahluwalia

I have a similar issue while pulling any repository
Details - Ubuntu 14.04.3 LTS
Docker installed using https://blog.docker.com/2015/07/new-apt-and-yum-repos/
Error Message -
8916b0188c16: Error pulling image (1.0.1) from weaveworks/weaveexec, endpoint: https://registry-1.docker.io/v1/, Error mounting '/dev/mapper/docker-202:2-492994-511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' on '/var/lib/8916b0188c16: Error pulling image (1.0.1) from weaveworks/weaveexec, Error mounting '/dev/mapper/docker-202:2-492994-511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' on '/var/lib/docker/devicemapper/mnt/511136ea3c5a64f264b7Error pulling image (1.0.1) from weaveworks/weaveexec, Error mounting '/dev/mapper/docker-202:2-492994-511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' on '/var/lib/docker/devicemapper/mnt/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158': invalid argument

demo@demo:~$ docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

demo@demo:~$ docker info
Containers: 0
Images: 60
Storage Driver: devicemapper
Pool Name: docker-202:2-492994-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 13.67 GB
Data Space Total: 107.4 GB
Data Space Available: 68.91 GB
Metadata Space Used: 10.15 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.137 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-55-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 4
Total Memory: 15.67 GiB

This is happening to any pull or start of container.....

@dcosson
dcosson commented Aug 6, 2015

Just to clarify, are the new apt & yum repos recommended for production use at this point? The docs still say to install via wget -qO- https://get.docker.com/ | sh.

@mbentley
Contributor
mbentley commented Aug 6, 2015

Yes, the get.docker.com script points to the new apt & yum repos.

@IPvFletch

I'm using the Amazon ECS AMI, and when I remove docker and re-install via the above script (which just runs yum -y -q install docker) it installs the same version I've had, 1.6.2. I guess the AMZN repo doesn't have a new docker version yet?

I did however manage to get the repo for centos7 and get it installed (on top of docker 1.6.2 using --force and --nodeps) and with ecs-init still installed for ECS. It seems to be working too! :D

@shredder12

I am running Docker 1.8.1 installed from the new repos with Udev Sync support, but unfortunately I am facing a similar issue on reboot.

$ docker --version
Docker version 1.8.1, build d12ea79

$ docker info
Containers: 4
Images: 48
Storage Driver: devicemapper
Pool Name: docker-8:3-16335425-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 3.17 GB
Data Space Total: 107.4 GB
Data Space Available: 104.2 GB
Metadata Space Used: 3.867 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.144 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-62-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 8
Total Memory: 7.772 GiB
Name: trusty-test
ID: EYRP:YG6F:FM66:BFSG:NQM3:LZDY:VIG2:EF3X:RVPW:5DHL:BDF6:OBUC

$ uname -a
Linux trusty-test 3.13.0-62-generic #102-Ubuntu SMP Tue Aug 11 14:29:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Scenario:

  • Host is running 4 containers with host mapped volumes.
  • Host is rebooted.
  • 'docker start' for some container returns filesystem issue, while for others it returns
    Error response from daemon: Cannot start container container2: Error getting container 0461efa637267d890975d5953085051837d3f74f37b3f99a8b5ad07a98147738 from driver devicemapper: Unknown device 0461efa637267d890975d5953085051837d3f74f37b3f99a8b5ad07a98147738

Sometimes its 'unknown device' instead of unknown filesystem.
Error response from daemon: Cannot start container container1: Error getting container d5253dfb678d4f528dda97072a653515ec0ac58ab576107b06bb7075a8475474 from driver devicemapper: Unknown device d5253dfb678d4f528dda97072a653515ec0ac58ab576107b06bb7075a8475474

Workaround:

service docker stop && rm -r /var/lib/docker && service docker start

@shadeslayer shadeslayer added a commit to blue-systems/pangea-tooling that referenced this issue Oct 6, 2015
@shadeslayer shadeslayer Keep retrying till the container starts since there appears to
be a race condition in udev.

Refer to docker/docker#4036
8229a33
@PhilibertDugas

Facing a similar issue trying to do a docker run on my image,

The problem occurs on a Redhat 7.1 installation but works fine on a Ubuntu 14.04 server.

# docker --version
Docker version 1.7.1, build 446ad9b/1.7.1

# docker info
Containers: 324
Images: 183
Storage Driver: devicemapper
 Pool Name: docker-253:6-257148-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 4.371 GB
 Data Space Total: 107.4 GB
 Data Space Available: 98.33 GB
 Metadata Space Used: 20 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.127 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.14.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.1 (Maipo)
CPUs: 8
Total Memory: 31.26 GiB
Name: 
ID: 7UQC:JCTV:FCZA:EKNT:HMX4:BRDX:DU5L:MNGX:7EWW:XUFQ:J2QT:MCN2

# uname -a
Linux 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Aug 25 11:21:22 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

Still not sure what to do about it, it occurs > 50% of the time

docker-issue

@madeleineth

@PhilibertDugas Are you able to upgrade to Docker 1.8.3? I did that, and the problem went away for me.

@PhilibertDugas

I tried as well with Docker 1.8.3 but I face the same issue

@PhilibertDugas

Changing from Device Mapper to OverlayFS did the trick for me!

@MarcoLotz

having the same problem on docker 1.9.1
Not common tho. Happens once every 10 started containers. The containers are all nginx.

on /var/log/upstart/docker.log:

INFO[32197] POST /v1.21/containers/4d2190e5f5fa/start

DEBU[32198] Assigning addresses for endpoint evil_brattain's interface on network labs6

DEBU[32198] RequestAddress(LocalDefault/172.24.0.0/16, , map[])

DEBU[32198] /sbin/iptables, [--wait -t nat -A DOCKER -p tcp -d 0/0 --dport 36669 -j DNAT --to-destination 172.24.1.66:80 ! -i br-10febd436870]

DEBU[32199] /sbin/iptables, [--wait -t filter -A DOCKER ! -i br-10febd436870 -o br-10febd436870 -p tcp -d 172.24.1.66 --dport 80 -j ACCEPT]

DEBU[32201] /sbin/iptables, [--wait -t nat -A POSTROUTING -p tcp -s 172.24.1.66 -d 172.24.1.66 --dport 80 -j MASQUERADE]

DEBU[32203] Assigning addresses for endpoint evil_brattain's interface on network labs6

DEBU[32203] /sbin/iptables, [--wait -t nat -D DOCKER -p tcp -d 0/0 --dport 36669 -j DNAT --to-destination 172.24.1.66:80 ! -i br-10febd436870]

DEBU[32204] /sbin/iptables, [--wait -t filter -D DOCKER ! -i br-10febd436870 -o br-10febd436870 -p tcp -d 172.24.1.66 --dport 80 -j ACCEPT]

DEBU[32206] /sbin/iptables, [--wait -t nat -D POSTROUTING -p tcp -s 172.24.1.66 -d 172.24.1.66 --dport 80 -j MASQUERADE]

DEBU[32207] Releasing addresses for endpoint evil_brattain's interface on network labs6

DEBU[32207] ReleaseAddress(LocalDefault/172.24.0.0/16, 172.24.1.66)

ERRO[32210] Handler for POST /v1.21/containers/4d2190e5f5fa/start returned error: Cannot start container 4d2190e5f5fa: mounting mqueue mqueue : device or resource busy

ERRO[32210] HTTP Error err=Cannot start container 4d2190e5f5fa: mounting mqueue mqueue : device or resource busy statusCode=500

Didn't try with OverlayFS yet.

@thaJeztah
Member

@MarcoLotz can you open a new issue? I've seen those messages as well occasionally, perhaps its already being worked on, but I think it's a different issue than the one being discussed here

@slampenny

In case anyone gets a similar message.... I got it when the script that brings docker up tried to mount a directory that wan't there.

For instance:

docker run -d -v /Users/jordan/Code:/home/docker -p 8000:80 -p 2222:22 -p 4430:443 --name web --link mongo:mongo --link redis:redis MY_IMAGE

On the box to which I was deploying, /Users/jordan/Code didn't exist, because it was an ubuntu box, not an OSX box.

@liusdu
Contributor
liusdu commented Dec 1, 2015

I fix a devicemapper issue like the folowing in. #18329

$ docker rm -f fdca994bec21
Error response from daemon: Driver devicemapper failed to remove root filesystem fdca994bec2100335c912d56540acc6df271b36b2554270c606836277e159e15: Device is Busy
Error: failed to remove containers: [fdca994bec21]

If you have running containers in aufs/devicemaper and the daemon crashed. When you restart the daemon and attempt to remove the remaining containers. You will get this error.

@slatkovic

Switching from devicemapper to aufs solved the problem for me.

@markuskhouri markuskhouri referenced this issue in CoderDojoTC/python-minecraft Dec 23, 2015
Open

need newer docker version? #22

@AaronDMarasco-VSI

I hate to be participant 132 in a huge thread, but after skimming this ticket, I think I "should" be OK, but am not. Using CentOS7:

$ docker info | grep Storage
Storage Driver: devicemapper
$ docker info |grep -i "udev.*sync"
 Udev Sync Supported: true
$ docker info | grep -i loop
(no response)

This is an example of the error I receive:

docker run -d -v /opt/Xilinx/:/opt/Xilinx/:ro -v /data/jenkins_workspaces/dds:/build --name jenkins-dds-460 jenkins/build:v2-C6
5eadc30da8d1f60a13db3392ba751b197079c579a3dbba43f6a092269a475320
Error response from daemon: Cannot start container 5eadc30da8d1f60a13db3392ba751b197079c579a3dbba43f6a092269a475320: Error getting container 5eadc30da8d1f60a13db3392ba751b197079c579a3dbba43f6a092269a475320 from driver devicemapper: open /dev/mapper/docker-253:6-101737821-5eadc30da8d1f60a13db3392ba751b197079c579a3dbba43f6a092269a475320: no such file or directory

This happens 5-20% of the time it seems.

A few days ago, I had Docker totally trash all its containers. I researched some, and learned that on CentOS 7 hosts, the preferred method is a "raw" LVM thin provisioned, so that's what I did:

$ cat /etc/systemd/system/docker.service.d/0-move-library.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon --exec-opt native.cgroupdriver=cgroupfs -H fd:// --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/vg_ex-docker--pool --storage-opt dm.use_deferred_removal=true

# Needs newer systemd than CentOS uses: https://bugzilla.redhat.com/show_bug.cgi?id=1200946
# --storage-opt dm.use_deferred_deletion=true

I'm not listing here all the LVM stuff I did, but it was working for a few days...

docker info

$ cat /etc/redhat-release 
CentOS Linux release 7.1.1503 (Core) 

$ uname -a
Linux redacted.example.com 3.10.0-229.20.1.el7.x86_64 #1 SMP Tue Nov 3 19:10:07 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ docker --version
Docker version 1.9.1, build a34a1d5

$ docker info
Containers: 9
Images: 35
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: vg_ex-docker--pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 20.51 GB
 Data Space Total: 322.1 GB
 Data Space Available: 301.6 GB
 Metadata Space Used: 14.26 MB
 Metadata Space Total: 4.001 GB
 Metadata Space Available: 3.987 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.20.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 56
Total Memory: 251.6 GiB
@AaronDMarasco-VSI

It definitely seems to be a race condition, when my Jenkins tries to spawn a handful of containers at once. I manually tried this on the server:

for i in $(seq 1 25); do
  docker run -d -v /opt/Xilinx/:/opt/Xilinx/:ro -v /data/jenkins_workspaces/examples:/build --name jenkins-examples-485 jenkins/ocpibuild:v2-C6; 
  echo $?; 
  docker rm -f jenkins-examples-485; 
done

And it ran flawlessly. However, if I background them instead, to force all 25 to try to create at once:

for i in $(seq 1 25); do 
  docker run -d -v /opt/Xilinx/:/opt/Xilinx/:ro -v /data/jenkins_workspaces/examples:/build --name jenkins-ctests-474-${i} jenkins/ocpibuild:v2-C6 & 
  echo $?; 
done
b66c5c2276b40854fc9f7b21e6bf153d861a2493638fb277195ad6a9376f8d32
621dc1b143f1a8728fb93a8227231a7b4488df0d96368e65d21956a6d6272004
6c6d1b417cd343469804c16f4d8479775f3aba905014d72cdb061f141f2ba816
127fdfa18b8d332da7a26e9cd9cb2f49ee4d4a000cf7ea2fe20858fffe1d7a09
66c0100da826fb6a94c084b3f4d9c53583b3e699e8a26076035c0f09bf61ec30
d16237d791da495d711483dc02c7021ab42781c001eb46128a064b3f1ddb245e
72176e1075e4a837ae7abeee9bfaee30b2f08f27cfa3542a66ebcd6a51115326
7d10de31239df16bca6c69d31d151d5ad7f5e44036f98e394e14fa9853408824
a582799ff489507fd13b02b3e2f2032a93c7f1bfbc88d443fee4526b8bddf84d
84de8f525c76ca1b64ea0c25f63a885a0e35ffa0cf2493361c56270635149023
0f104d795cb386542f0850bf964a5f426fe01476d8b56a414fdd98c6f145a973
1ad7c18178d54a53fa4fa4cda67de75923af1c965527a713542f73ac8a58cc52
ce153bb43b1fdb4ad193e72d36e8451baedf97c8c81bbe18f19818f7492eaf74
29ccbf74238de02c96eeee1eabd2f45e9163ab0633d96465b76d10a83e81f947
f178c90fb051303180bfcd0cde370526bd4a96f37ed94b3bcf56350d989e267f
46e91895d8c5c475bee7db2dc1a4c0592d4a715628dce807eff353739c7aeb4e
4c36736d89d06d62509a70203c66b74dd6ce37e7de6e6c5a2ffb3174525eed0d
86a913665de1522922726e901a54800414cc9858679168228fa626a255704d4c
c38da55732a1ff5df62238391b042a2129ad701ab2ddb826f99c81326bf26f74
57b6e0356ca25b3746c680bd6a535c514d64f4bcb50b900b9a93018207034b72
7651b72be21fb358b3fc01bd006b6e0c8fa8eb958e1cb1719e9ac223e214770e
521cc795b9e9f65cc5651a457593b540369188e17281657488490e36b62f370a
Error response from daemon: Cannot start container 7651b72be21fb358b3fc01bd006b6e0c8fa8eb958e1cb1719e9ac223e214770e: Error getting container 7651b72be21fb358b3fc01bd006b6e0c8fa8eb958e1cb1719e9ac223e214770e from driver devicemapper: open /dev/mapper/docker-253:6-101737821-7651b72be21fb358b3fc01bd006b6e0c8fa8eb958e1cb1719e9ac223e214770e: no such file or directory
Error response from daemon: Cannot start container 521cc795b9e9f65cc5651a457593b540369188e17281657488490e36b62f370a: Error getting container 521cc795b9e9f65cc5651a457593b540369188e17281657488490e36b62f370a from driver devicemapper: open /dev/mapper/docker-253:6-101737821-521cc795b9e9f65cc5651a457593b540369188e17281657488490e36b62f370a: no such file or directory
ac57455b5030d56f163b45e7d1f05856c0429823fc708b37aa21b5925f70da37

So 2 of 25 failed.

And they "stay dead":

$ docker ps -a | grep 521cc795b9
521cc795b9e9        jenkins/ocpibuild:v2-C6   "/bin/sleep 1h"     2 minutes ago       Created                                       jenkins-ctests-474-3
$ docker start jenkins-ctests-474-3
Error response from daemon: Cannot start container jenkins-ctests-474-3: Error getting container 521cc795b9e9f65cc5651a457593b540369188e17281657488490e36b62f370a from driver devicemapper: open /dev/mapper/docker-253:6-101737821-521cc795b9e9f65cc5651a457593b540369188e17281657488490e36b62f370a: no such file or directory
Error: failed to start containers: [jenkins-ctests-474-3]
@thaJeztah
Member

@AaronDMarasco-VSI @RoelVdP could you open a new issue instead? (but feel free to link to this issue). This discussion is already really long, and I think it's better to start a "fresh" one.

@EwanValentine

I'm getting...

Recreating 051010d58e_051010d58e_051010d58e_051010d58e_051010d58e_051010d58e_051010d58e_051010d58e_051010d58e_bandzest_core-api_1
ERROR: Cannot start container 40dcfe9579ccce5fceb7bff3d1e1c33325f6b1b64d0b532e1b0dc9fbf2958274: [8] System error: no such file or directory

For no apparent reason. I'm guessing this is related? I'm trying to run a golang app in Docker on Ubuntu 14.04.

@thaJeztah
Member

@EwanValentine looks not related to this one; think that's this one #18098

@EwanValentine

I'm not sure how they're related @thaJeztah there doesn't seem to be a suggested fix or definitive cause in that reference :(

@thisiswangle

Similar error while trying to build with docker 1.9.0 and 1.9.1:

$ docker build -t dev/base:v1.0 .
Sending build context to Docker daemon 4.608 kB
Step 1 : FROM ubuntu:14.04
 ---> ca4d7b1b9a51
Step 2 : MAINTAINER Le Wang "thisiswangle@gmail.com"
 ---> Using cache
 ---> 1c99faca4ce3
Step 3 : RUN dpkg-divert --local --rename --add /sbin/initctl
 ---> Using cache
 ---> 61530fa5a55e
Step 4 : RUN ln -sf /bin/true /sbin/initctl
 ---> Using cache
 ---> 21bc3d4993b4
Step 5 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> 3b9f5744dc3c
Step 6 : COPY sources.list.trusty /etc/apt/sources.list
Error getting container 9901a9ff6b11246b08ff5be40848540a544b89ea4317467915b70005909bc26b from driver devicemapper: Error mounting '/dev/mapper/docker-8:1-138421-9901a9ff6b11246b08ff5be40848540a544b89ea4317467915b70005909bc26b' on '/var/lib/docker/devicemapper/mnt/9901a9ff6b11246b08ff5be40848540a544b89ea4317467915b70005909bc26b': no such file or directory

Building process works well after a few times retry.

Here is my docker info:

$ docker info
Containers: 3
Images: 33
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-8:1-138421-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem:
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 3.434 GB
 Data Space Total: 107.4 GB
 Data Space Available: 36.98 GB
 Metadata Space Used: 3.65 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Udev Sync Supported: false
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-66-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 2
Total Memory: 1.955 GiB
Name: vagrant-ubuntu-trusty-64
ID: WI7O:J45C:QK3V:IYCB:TCVO:SWNY:SFRI:4C3V:INOB:QNGO:67E6:JLCK

My Host OS is OSX EI Capitan 10.11.2 + Virtual Box 5.10 + Vagrant 1.7.4.
My Guest OS is Ubuntu 14.04 x86_64.

@andrecp
andrecp commented Feb 3, 2016

@thisiswangle I too have the same problem my setup is
virtualbox 5.0.12
Vagrant 1.8.1
el captain 10.11.2

running on guest os Ubuntu 14.04 x86_64 3.13.0-77-generic

@xarem
xarem commented Feb 4, 2016

I'm getting this error every 2-3 builds... i have to restart the command and then it works.

$ docker build -t .
Sending build context to Docker daemon 557.1 kBSending build context to Docker daemon 1.114 MBSending build context to Docker daemon 1.639 MBSending build context to Docker daemon 2.195 MBSending build context to Docker daemon 2.753 MBSending build context to Docker daemon  3.31 MBSending build context to Docker daemon 3.867 MBSending build context to Docker daemon 4.424 MBSending build context to Docker daemon 4.981 MBSending build context to Docker daemon 5.538 MBSending build context to Docker daemon 6.095 MBSending build context to Docker daemon 6.652 MBSending build context to Docker daemon 7.209 MBSending build context to Docker daemon 7.766 MBSending build context to Docker daemon 8.323 MBSending build context to Docker daemon  8.88 MBSending build context to Docker daemon 9.437 MBSending build context to Docker daemon 9.994 MBSending build context to Docker daemon 10.55 MBSending build context to Docker daemon 11.11 MBSending build context to Docker daemon 11.67 MBSending build context to Docker daemon 12.22 MBSending build context to Docker daemon 12.75 MBSending build context to Docker daemon  13.3 MBSending build context to Docker daemon 13.86 MBSending build context to Docker daemon 14.39 MBSending build context to Docker daemon 14.94 MBSending build context to Docker daemon  15.5 MBSending build context to Docker daemon 16.06 MBSending build context to Docker daemon 16.61 MBSending build context to Docker daemon 17.15 MBSending build context to Docker daemon 17.69 MBSending build context to Docker daemon 18.25 MBSending build context to Docker daemon 18.81 MBSending build context to Docker daemon 19.37 MBSending build context to Docker daemon 19.92 MBSending build context to Docker daemon 20.48 MBSending build context to Docker daemon 21.04 MBSending build context to Docker daemon 21.59 MBSending build context to Docker daemon 22.15 MBSending build context to Docker daemon 22.38 MBSending build context to Docker daemon 22.38 MB
Step 1 : FROM whatwedo/symfony3:latest
latest: Pulling from whatwedo/symfony3
Digest: sha256:45d451017fb77b24c5f8357563d8ae7562197c11c28fd63c3c356179ff9ebf72
Status: Downloaded newer image for whatwedo/symfony3:latest
 ---> 1d5f72b7a6d2
Step 2 : ADD . /var/www
 ---> cc138333ab5a
Removing intermediate container 893287b5028e
Step 3 : WORKDIR /var/www
 ---> Running in b66ce2b959ff
 ---> c2c230788706
Removing intermediate container b66ce2b959ff
Step 4 : RUN cp /var/www/app/config/parameters.yml.docker /var/www/app/config/parameters.yml
 ---> Running in afeaff032b7f
 ---> 0881dbc9d17a
Removing intermediate container afeaff032b7f
Step 5 : RUN cp /var/www/app/config/nginx/nginx.conf /etc/nginx
 ---> Running in 16f015be3f2c
 ---> 1398471886d0
Removing intermediate container 16f015be3f2c
Step 6 : RUN apt-get update && apt-get install xvfb xfonts-75dpi fontconfig libxrender1 pdftk -qq
Error getting container 1312109b05a8c73346f481e1b1533e3cf030cd0ab5d7914d811b168745a86105 from driver devicemapper: Error mounting '/dev/mapper/docker-253:1-1318774-1312109b05a8c73346f481e1b1533e3cf030cd0ab5d7914d811b168745a86105' on '/var/lib/docker/devicemapper/mnt/1312109b05a8c73346f481e1b1533e3cf030cd0ab5d7914d811b168745a86105': no such file or directory

ERROR: Build failed with: exit status 1
root@gci01:~# docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64
root@gci01:~# docker info
Containers: 6
Images: 124
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-253:1-1318774-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: 
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 5.76 GB
 Data Space Total: 107.4 GB
 Data Space Available: 19.54 GB
 Metadata Space Used: 9.286 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.138 GB
 Udev Sync Supported: false
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-71-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 1
Total Memory: 994 MiB
Name: gci01
ID: 6ARE:3PQL:I452:ADT2:TQXI:S6KG:6W5N:YDFH:NXUZ:OIOF:BPRA:DOV2
WARNING: No swap limit support
root@gci01:~# uname -r
3.13.0-71-generic

on DigitalOcean 1GB instance

@andrecp
andrecp commented Feb 4, 2016

What I've done to fix it (apparently it did) was installing the docker-engine instead of the docker-lxc in the ubuntu machine I have...

I think docker-lxc is probably legacy and shouldn't be used.

In the docker-engine installation I don't get

Udev Sync Supported: false

@thaJeztah
Member

@xarem how did you install docker; did you use the apt repository, or install a static binary? I see you're using devicemapper without Udev sync support; it's strongly discouraged to run without Udev sync, as that will lead to data loss, and strange behavior like this.

The default storage driver for Ubuntu is aufs, which will be used if you install using the installation procedure in https://docs.docker.com/engine/installation/ubuntulinux/

You will need to wipe your /var/lib/docker to do a fresh install though (otherwise the devicemapper dir will be still there)

@grisha87
grisha87 commented Feb 4, 2016

++@andrecp was about to post the same answer. Installing the latest "docker-engine" from the repository actually fixes this issue. What's important here is that you need to have all the extra linux kernel features installed (so called extras) prior installing docker-engine. Also the key to success is to get the dynamically linked docker binary:

$ ldd /usr/bin/docker 
        linux-vdso.so.1 =>  (0x00007ffd273fe000)
        libesets_pac.so => /usr/lib/libesets_pac.so (0x00007f557ff99000)
        libapparmor.so.1 => /usr/lib/x86_64-linux-gnu/libapparmor.so.1 (0x00007f557fd8d000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f557fb6f000)
        libdevmapper.so.1.02.1 => /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 (0x00007f557f936000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f557f571000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f557f36c000)
        /lib64/ld-linux-x86-64.so.2 (0x000055a1a2fb5000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f557f149000)
        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f557ef38000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f557ecf9000)
        libcgmanager.so.0 => /lib/x86_64-linux-gnu/libcgmanager.so.0 (0x00007f557eade000)
        libnih.so.1 => /lib/x86_64-linux-gnu/libnih.so.1 (0x00007f557e8c6000)
        libnih-dbus.so.1 => /lib/x86_64-linux-gnu/libnih-dbus.so.1 (0x00007f557e6bb000)
        libdbus-1.so.3 => /lib/x86_64-linux-gnu/libdbus-1.so.3 (0x00007f557e476000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f557e26e000)

Above results with enabled udev sync support for devicemapper.

@ZYNCMA
ZYNCMA commented May 17, 2016 edited

I ran for i in {0..100}; do docker run busybox echo test; done using docker 1.11 and got no error.
Can I say that devicemapper is race-free using docker 1.11?
If I can, what differences does udev_sync_supported make?

docker info (sorry that I can't publish kernel and os)

# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 1.11.0
Storage Driver: devicemapper
 Pool Name: docker-8:4-1076834816-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 4.448 GB
 Data Space Total: 107.4 GB
 Data Space Available: 102.9 GB
 Metadata Space Used: 3.617 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Udev Sync Supported: false
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /data/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /data/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: host null
Kernel Version: 
Operating System: 
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 62.65 GiB
Name: 
ID: 2UZG:RFJS:7SVS:3RE5:H7XK:6Q4N:TVOO:KZND:SDEF:POSS:ZEWF:ESAO
Docker Root Dir: /data/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
@davehodg

This is still a problem? I have:

docker --version

Docker version 1.11.1, build 5604cbe

Getting:

docker build -t mongodb .

Sending build context to Docker daemon 16.9 kB
Step 1 : FROM registry.access.redhat.com/rhel7.2
---> bf2034427837
Step 2 : MAINTAINER Joe Bloggs
devmapper: Unknown device b4904eb2bd71c7aa6cc50c8ac7c9695845641de531ed6d8b97742857258b418e

docker version

Client:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Wed Apr 27 00:34:42 2016
OS/Arch: linux/amd64

Server:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Wed Apr 27 00:34:42 2016
OS/Arch: linux/amd64

So pretty fresh I think.

@thaJeztah
Member

@davehodg can you show your docker info?

@davehodg

docker info

Containers: 32
Running: 0
Paused: 0
Stopped: 32
Images: 1
Server Version: 1.11.1
Storage Driver: devicemapper
Pool Name: docker-253:0-20487-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 11.8 MB
Data Space Total: 107.4 GB
Data Space Available: 11.93 GB
Metadata Space Used: 581.6 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-12-01)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge null
Kernel Version: 3.10.0-327.13.1.el7.x86_64
Operating System: Red Hat Enterprise Linux
OSType: linux
Architecture: x86_64
CPUs: 3
Total Memory: 5.518 GiB
Name: daves-macbook-air
ID: MIBZ:OB6L:J4Q2:OD6P:EVEZ:XEHS:JT3V:SLZG:ALFZ:NC47:IYXQ:72EE
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

@davehodg

If I were to get the freshest most up to date docker, where would I get it from?

@davehodg

OK, got the source from here. Tried make:

make

docker build -t "docker-dev:master" -f "Dockerfile" .
Sending build context to Docker daemon 130.2 MB
Step 1 : FROM debian:jessie
jessie: Pulling from library/debian
8b87079b7a06: Already exists
a3ed95caeb02: Already exists
Digest: sha256:32a225e412babcd54c0ea777846183c61003d125278882873fb2bc97f9057c51
Status: Downloaded newer image for debian:jessie
---> bb5d89f9b6cb
Step 2 : RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61 || apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61
devmapper: Unknown device 50a33b69e5edbf78d012da68ac249b1af5a9fddaeb316c662e6a65f763c974e2
make: *** [build] Error 1

I think I might make a new RHEL7 vmware image...

@thaJeztah
Member

@davehodg also see #22031

@davehodg

Not really telling me anything new :(

@thaJeztah
Member

@davehodg no, problem is indeed that it's quite an unpredictable issue. It's known to be problematic on systems without udev-sync, also running on "loop devices" doesn't help.

@davehodg

Thanks. I'm so far down the rabbit hole at this point.

@davehodg

Register my whine, but I've moved to running Docker on my Mac. Let's see how that transpires.

@jizhilong
Contributor

I also encountered this issue in our environment, and I worked out a way to reproduce it, hope it may help to resolve the issue.

docker info

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.11.1
Storage Driver: devicemapper
 Pool Name: docker-252:1-58724678-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 883.9 MB
 Data Space Total: 107.4 GB
 Data Space Available: 35.03 GB
 Metadata Space Used: 2.146 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.145 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.86 GiB
Name: jzl-dev0010
ID: U4ZX:62TO:YRGL:CNDH:6UZN:VNYX:3R2W:ZCWF:4TJS:54BS:O4AK:NLYY
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/

way to reproduce

make sure all containers removed

# NOTE!!! this command is dangerous, don't execute it on any non-experimental node.
docker rm -f  `docker ps -aq`

get image ready

a busybox image would be sufficient, docker pull busybox

get script ready

A Python script is needed to eagerly detect and open any docker created non-init dm devices:

import os
import time

while True:
    try:
        for line in os.listdir('/dev/mapper'):
            if 'init' in line:continue
            if len(line) >= 80:
                filename=os.path.join('/dev/mapper/',line)
                f = open(filename)
        print 'got it %s' % filename
        time.sleep(3*10**7)
    except Exception:
        pass

make sure all container or image related dm devices removed

run dmsetup info to list all dm devices on your node, and run dmsetup remove <name> to remove all devices whose name starts with docker- and ends with a uuid or <uuid>-init, such as docker-252:1-58724678-3c96b9d15c41b49816eab1faaa0cb6eaa7cdffe5decd45f2bd48ae72f32e16f5

run the script and reproduce the issue

following is one sample of my reproducing logs:

# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
# dmsetup info
Name:              docker-252:1-58724678-pool
State:             ACTIVE
Read Ahead:        8192
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 0
Number of targets: 1

# python opendm.py &
[1] 3757
# cid=`docker create -it busybox sh`
got it /dev/mapper/docker-252:1-58724678-097a341a6a5f17cc86f9d83ebdff35b8da86fd8ab6d614ac8445f3fabe1d6e25
# docker diff $cid
# docker start $cid
Error response from daemon: open /dev/mapper/docker-252:1-58724678-097a341a6a5f17cc86f9d83ebdff35b8da86fd8ab6d614ac8445f3fabe1d6e25: no such file or directory
Error: failed to start containers: d3cbd8f86704136511b5d4cfdafeb1398a46f962ff9af7389ec7368f8472433e

the essential commands are:

python opendm.py &
docker diff $cid
docker start $cid

get the corrupted container back to normal

# kill %1
# dmsetup remove docker-252:1-58724678-097a341a6a5f17cc86f9d83ebdff35b8da86fd8ab6d614ac8445f3fabe1d6e25
[1]+  Terminated              python opendm.py
# ls /dev/dm-*
/dev/dm-0
# ls /dev/mapper/
control  docker-252:1-58724678-pool
# docker start $cid
d3cbd8f86704136511b5d4cfdafeb1398a46f962ff9af7389ec7368f8472433e

the essential operations to recover the container include:

  1. kill the background process python opendm.py
  2. remove the container's dm device.
@thaJeztah
Member

@jizhilong can you please open a new issue? The original issue reported here was resolved and related to udev-sync not being available, causing corruption. Since your machine looks to have udev-sync support, it's likely a different issue. This discussion is already very long, so it's better to open a new issue, than commenting on a closed issue.

@thaJeztah
Member
thaJeztah commented Jul 14, 2016 edited

I'm locking this issue, and provide a link and quote to the resolution for the issue that was originally reported: #4036 (comment). If you still encounter this, please open a new issue with as much information as possible and steps to reproduce.

(I modified some outdated information in the description below)

This issue is resolved

Key item to observe before commenting on this issue, is whether docker info | grep Udev returns for Udev Sync Supported: true or not.

Udev sync

Devicemapper storage driver expects to be synchronized with udev. When this info reports false, that means that devicemapper and udev are not able to sync. In this event there is a race condition that many of you have experienced.

Causes

There are a couple of causes for Udev sync to not be supported:

  • The docker binary used on your host, was statically linked when compiled, instead of dynamically linking.
  • If your docker binary is dynamically linked, the version of libdevmapper.so too old to support Udev sync

Solutions

  1. Install the docker binary from our apt/yum repositories; these binaries are dynamically linked
  2. Build the docker binary yourself, dynamically linked
    • git clone git://github.com/docker/docker.git
    • cd docker
    • AUTO_GOPATH=1 ./hack/make.sh dynbinary
  3. If your docker binary is dynamic (test with file $(which docker) | grep dynamically) and Udev sync reports false, you will likely need to update the distribution release used

Docker 1.11 and up will refuse to start the daemon if Udev sync is not supported (see #21097)

@thaJeztah thaJeztah locked and limited conversation to collaborators Jul 14, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.