Devicemapper "Unknown Device" when trying to start "Created" container

[andrewmichaelsmith]

Output of docker version:

Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 06:06:01 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 06:06:01 UTC 2015
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 74
Images: 2679
Storage Driver: devicemapper
 Pool Name: docker-docker--pool
 Pool Blocksize: 524.3 kB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 74.29 GB
 Data Space Total: 161.7 GB
 Data Space Available: 87.44 GB
 Metadata Space Used: 88.63 MB
 Metadata Space Total: 406.8 MB
 Metadata Space Available: 318.2 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Library Version: 1.02.107-RHEL7 (2015-12-01)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 8
Total Memory: 7.549 GiB
Name: box
ID: EUW6:ACRN:RAHK:FCIT:JO2E:NMNN:OV3M:RNEY:JNAZ:X2VC:MLXX:WXNF
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):

• Physical machine (and AWS)
• CentOS 7
• Linux 3.10.0-327.10.1.el7.x86_64 Container tagging / naming #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Devicemapper

Steps to reproduce the issue:

We see this issue quite often but I've failed to reproduce it consistently. If you could provide us with any pointers we'd be very grateful.

Describe the results you received:

It is sometimes the case that a container is "Created" (using, I think, docker create) but a docker start of that container fails with an error:

$ sudo docker ps -a | grep Created
217410d0ab96        registry/google_containers/pause:2.0   "/pause"                 34 seconds ago       Created                                        happy_engelbart
$ sudo docker start 7a572aafa393
Error response from daemon: Cannot start container 217410d0ab96: Error getting container 217410d0ab9622715cb9db4a7ba34439c3e34a3683e685855b979894768ada43 from driver devicemapper: Unknown device 217410d0ab9622715cb9db4a7ba34439c3e34a3683e685855b979894768ada43
Error: failed to start containers: [217410d0ab96]

As you might expect, this is not showing up in devicemapper:

$ lsblk | grep 217410d0ab96 $

It looks like - even though docker thinks the container exists, devicemapper does not.

Describe the results you expected:

I would expect 217410d0ab96 to start without issue.

Additional information you deem important (e.g. issue happens only occasionally):

Unfortunately the issue only happens occasionally. But often enough that we will usually have a few lying around, I'm ready and waiting to provide any additional debug you need.

[andrewmichaelsmith]

Is there any additional information I could provide to help diagnose this issue?

[andrewmichaelsmith]

We saw this issue again today, docker complaining about "Unknown device":

Apr 28 17:05:05 box-2 kubelet[2755]: E0428 17:05:05.335856    2755 generic.go:229] PLEG: Ignoring events for pod application-2-4p90a/ci: API error (500): Unknown device c14ab46b2f7132324eb3732e4e962f80f9774616a08e477f7b97d76a865a4903

docker ps thinks the container exists:

[17:05:05] company@box-2:~$ sudo docker ps -a | grep c14ab
c14ab46b2f71        docker.registry.network.company/google_containers/pause:2.0   "/pause"                 5 hours ago         Created                                      k8s_POD.4ace886_application-2-4p90a_ci_5aa4c382-0d2b-11e6-97e4-00163e1b3b80_a45865fe

docker inspect disagrees:

[17:13:06] root@box:/var/lib/docker/devicemapper# docker inspect c14ab46b2f7132324eb3732e4e962f80f9774616a08e477f7b97d76a865a4903
Error: No such image or container: c14ab46b2f7132324eb3732e4e962f80f9774616a08e477f7b97d76a865a4903
[]

If we take a look at the devicemapper files, there is 1 for the container, 1 init for mnt, 1 init for metadata:

[17:07:25] root@box-2:/var/lib/docker# find . -name 'c14ab4*'
./devicemapper/metadata/c14ab46b2f7132324eb3732e4e962f80f9774616a08e477f7b97d76a865a4903-init
./devicemapper/mnt/c14ab46b2f7132324eb3732e4e962f80f9774616a08e477f7b97d76a865a4903-init
./containers/c14ab46b2f7132324eb3732e4e962f80f9774616a08e477f7b97d76a865a4903

Having a look at these flies (not that this means anything to me):

[17:07:48] root@box-2:/var/lib/docker/devicemapper/metadata# cat c14ab46b2f7132324eb3732e4e962f80f9774616a08e477f7b97d76a865a4903-init
{"device_id":2985,"size":107374182400,"transaction_id":5486,"initialized":false}[17:07:53]

[17:08:21] root@box-2:/var/lib/docker/devicemapper/mnt# ll c14ab46b2f7132324eb3732e4e962f80f9774616a08e477f7b97d76a865a4903-init/
total 0

Check dmsetup:

[17:09:26] root@box-2:/var/lib/docker/devicemapper# dmsetup table | grep c14ab4

This container does not appear to be known to devicemapper but is still known to docker. As you can see it's been there for 5 hours but docker hasn't picked up that it doesn't exist. I can't find any errors in the docker logs relating to this (aside from the "Unknown device" error above).

Any pointers on any debug I can switch on for docker (or devicemapper?) greatly appreciated.

[thaJeztah]

ping @rhvgoyal any suggestions?

@andrewmichaelsmith are you still able to reproduce this on the current version of docker? There's been a lot of changes since 1.8, and we don't do patch releases for older releases

[thaJeztah]

closing, because we didn't get a reply, open a new issue if you're still having this on the current release

[x1957]

same problem

Jun 08 10:24:54 c3-sa-i2-20151229-buf023.bj kubelet[10884]: E0608 10:24:54.679099   10884 generic.go:238] PLEG: Ignoring events for pod pi-mj15v/default: Error response from daemon: {"message":"devmapper: Unknown device ac86e939cc728918a1c9a19bf3d3486e497b47f824fbef06d0a2c2f9ff80c4f0"}

[marksteward]

Just saw this on docker 1.12.1

[thaJeztah]

@marksteward Docker 1.12.1 or 17.12.1? Docker 1.12.1 reached end of life in March of last year, and has not been maintained since.

However without full details, and without a reproducible case, this may be difficult to address. If you do have a consistent way to reproduce this on a current version of Docker, please open a new issue instead.

[marksteward]

Yep, no problem - just updating here in case anyone stumbles across it in the future. We've got some theories about what caused it.