Devicemapper: Can't set cookie dm_task_set_cookie failed

[elliotpeele]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

More details can be found in: kubevirt/kubevirt#321

Expected behavior

Containers should start properly.

Actual behavior

Sometimes start tasks fail with the following error:
devmapper: Error activating devmapper device for '0a1bbd565d4619df84af2f06c85fe4dbda24fa0aecb4bd8ee7689f0238113ac2-init': devicemapper: Can't set cookie dm_task_set_cookie failed

Output of docker version:

Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:31:53 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.1-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 23:01:50 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.06.1-ce
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 20.45MB
 Data Space Total: 102GB
 Data Space Available: 102GB
 Metadata Space Used: 147.5kB
 Metadata Space Total: 1.07GB
 Metadata Space Available: 1.069GB
 Thin Pool Minimum Free Space: 10.2GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: qmzb64j5hhw75p3soz6xn80bn
 Is Manager: true
 ClusterID: pg1aaxo5e6iuwyng82qn9tdas
 Managers: 5
 Nodes: 17
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 10.24.7.221
 Manager Addresses:
  10.24.7.221:2377
  10.24.7.222:2377
  10.24.7.223:2377
  10.24.7.241:2377
  10.24.7.242:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-514.26.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 125.8GiB
Name: s01m01.example.com
ID: ADVG:AXW4:C3FB:4R7P:BAAG:PWIP:EGE5:4ZXT:2ZC4:2I4U:XJT7:WEF5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

I'm seeing this issue when attempting to launch a service in docker swarm on CentOS 7 using devicemapper storage driver on Docker CE 17.06.1.

docker service ps tenant1_rabbitmq

x8nueqqb1txj        tenant1_rabbitmq.2       docker.example.com/foo/rabbitmq:17w12   s01w04.example.com   Running             Running about an hour ago
8t1azr0cupq2         \_ tenant1_rabbitmq.2   docker.example.com/foo/rabbitmq:17w12   s01w06.example.com   Shutdown            Rejected about an hour ago   "devmapper: Error activating d…"
4qtslqtvvhlm         \_ tenant1_rabbitmq.2   docker.example.com/foo/rabbitmq:17w12   s01w06.example.com   Shutdown            Rejected about an hour ago   "devmapper: Error activating d…"
uw87lugpcxli         \_ tenant1_rabbitmq.2   docker.example.com/foo/rabbitmq:17w12   s01w06.example.com   Shutdown            Rejected about an hour ago   "devmapper: Error activating d…"
e5zqejda5n23         \_ tenant1_rabbitmq.2   docker.example.com/foo/rabbitmq:17w12   s01w06.example.com   Shutdown            Rejected about an hour ago   "devmapper: Error activating d…"

Each one of the failed tasks have an error like devmapper: Error activating devmapper device for '0a1bbd565d4619df84af2f06c85fe4dbda24fa0aecb4bd8ee7689f0238113ac2-init': devicemapper: Can't set cookie dm_task_set_cookie failed.

uname -a
Linux s01m01.example.com 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[ZeusF1]

Same issue on few containers
Built: Thu Aug 17 23:01:50 2017
3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[delfer]

label: area/storage
Same issue with docker 17.06.1-ce on CentOS 7 and loopback devicemapper
temporary workaround echo 'y' | sudo dmsetup udevcomplete_all
Docker 17.03.2-ce works fine

[yalinglee]

I got the same error on an AWS instance running CentOS 7.3.1611 and docker version 17.06.1-ce.
Linux 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Unfortunately, the errors didn't go away even after I ran echo 'y' | sudo dmsetup udevcomplete_all.
I'm still not able to remove the dead containers.

[romfreiman]

Any progress on this front? We ran into same (miss) behavior.

[Johnnei]

This PR docker/docker-ce#200 is likely related and is targeted for 17.06.2

[romfreiman]

@Johnnei what is the ETA for 17.06.2?

[Johnnei]

@rom-stratoscale I wish I knew

[hatdropper1977]

For a temporary work around until Docker fixes this, there are two ways to fix the issue.

If you want to keep using your existing server:
I fixed this issue by simply upgrading my Linux kernel from 3.10.0 to 4.4.83 (Current LT).

Background info:
I faced this very issue when I upgraded from Docker version 1.13.1-cs4, build e46aec0 to Docker version 17.06.1-ce, build 874a737

• Unmounting all docker volumes and then restarting the daemon provided a short lived fix
  • It worked for one run of docker-compose and then broke on the second
• Running the following command provided a temporary fix as well, I needed to run it before each docker_compose build.

$ echo 'y' | sudo dmsetup udevcomplete_all

• I initially upgraded the Linux kernel from 3.10.0 to 3.10.107. This did not fix the issue.

If you have the ability to destroy your server and start anew, you will be able to install Docker 17.06.1 without any problems on a fresh Centos 7 running 3.10.0. The issue only appears when you upgrade an existing server from Docker 13 to Docker 17. If you cannot wipe out your server and start anew, however, upgrading the Kernel fixes the issue.

[mlushpenko]

There is one more temporary solution - to increase semaphore limits (whatever that really means) by executing this command:

printf '250\t32000\t32\t300' >/proc/sys/kernel/sem

The last number is the limit. I tried first with 100 and docker worked few days fine, increasing number to 300 gave it 1-2 weeks of life. I think it depends on how often you destroy/restart containers. When you get that error again, just do echo 'y' | sudo dmsetup udevcomplete_all and you have another week or two.

I would put that semaphor limit to 10k, but don't really know how it affects the system except that it temporary solves the issue.

[henrygreg]

A slightly better solution that worked for me was to switch to aufs (or overlay) instead of devicemapper (https://docs.docker.com/engine/userguide/storagedriver/selectadriver/#check-and-set-your-current-storage-driver). This is assuming that you're fine with moving away from devicemapper. However, doing this will clear all the images.

[thaJeztah]

This should be resolved in docker 17.06.2; https://github.com/docker/docker-ce/releases/tag/v17.06.2-ce;

Devmapper: ensure UdevWait is called after calls to setCookie moby/moby#33732

I'm closing this issue because it should be resolved, but feel free to comment if you're still running into this on docker 17.06.2 or above

[OliverPereira]

I have just hit the same issue on the current version of Docker and I didn't have this problem yesterday.

Docker version 18.06.0-ce, build 0ffa825

Error starting daemon: error initializing graphdriver: devicemapper: Can't set cookie dm_task_set_cookie failed