Error response from daemon: devicemapper failed to remove the rootFS for <id> : devicemapper : Error running DeleteDevice dm_task_run failed

[ackris]

Pinging @rhvgoyal @thaJeztah

Details:
OS- CentOS Linux release 7.3.1611 (Core)
uname -a : 3.10.0-514.26.2.el7.x86_64

Docker version:

Client:
Version:      17.06.0-ce
API version:  1.30
Go version:   go1.8.3
Git commit:   02c1d87
Built:        Fri Jun 23 21:20:36 2017
OS/Arch:      linux/amd64

Server:
Version:      17.06.0-ce
API version:  1.30 (minimum version 1.12)
Go version:   go1.8.3
Git commit:   02c1d87
Built:        Fri Jun 23 21:21:56 2017
OS/Arch:      linux/amd64

Docker info:

Containers: 12
Running: 0
Paused: 0
Stopped: 12
Images: 14
Server Version: 17.06.0-ce
Storage Driver: devicemapper
Pool Name: docker-8:3-396359-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 5.607GB
Data Space Total: 107.4GB
Data Space Available: 7.345GB
Metadata Space Used: 7.221MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.14GB
Thin Pool Minimum Free Space: 10.74GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: q32938pqmetuskqk09afoyrrl
Is Manager: false
Node Address: 192.168.10.108
Manager Addresses:
  192.168.41.53:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
seccomp
  Profile: default
Kernel Version: 3.10.0-514.26.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 2.781GiB
Name: localhost.localdomain
ID: KQAH:NHUS:NOKK:KKKL:P62B:H5PC:NT6S:RSOG:D36X:UOUJ:JKW2:MSA2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: 2606
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Problem:
There are four containers in this node which are dead. I am unable to remove them.

Details:

docker rm -f <container_id> is not working

I browsed through GitHub issues for any solution. Instead I got an impression that this is not an error but a serious bug which is rearing its head again and again.

Help needed in resolving this issue.

Regards
Aditya

[ackris]

Hi @thaJeztah :)

I have referred issue number #26900. I don't want to delete /var/lib/docker... dir. Is there any other solution?

Regards
Aditya

[rhvgoyal]

Check state of thin pool. Paste journal messages here to see what failed. I suspect either device is busy or something is wrong with thin pool where its not allowing deleting devices. dmsetup table and dmsetup status might give some idea about thin pool state.

[ackris]

@rhvgoyal

[root@localhost docker]# dmsetup ls
docker-8:3-396359-2cb7a211f0b42095cfcef5485849bc8613cf0e1223ca3d29f1b3721988bf0f09      (253:1)
docker-8:3-396359-pool  (253:0)

[root@localhost docker]# dmsetup info
Name:              docker-8:3-396359-2cb7a211f0b42095cfcef5485849bc8613cf0e1223ca3d29f1b3721988bf0f09
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 1
Number of targets: 1

Name:              docker-8:3-396359-pool
State:             ACTIVE
Read Ahead:        8192
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1

[root@localhost docker]# dmsetup status
docker-8:3-396359-2cb7a211f0b42095cfcef5485849bc8613cf0e1223ca3d29f1b3721988bf0f09: 0 20971520 thin 6491904 20971519
docker-8:3-396359-pool: 0 209715200 thin-pool 323 1400/524288 70355/1638400 - rw discard_passdown queue_if_no_space –

[root@localhost docker]# dmsetup table
docker-8:3-396359-2cb7a211f0b42095cfcef5485849bc8613cf0e1223ca3d29f1b3721988bf0f09: 0 20971520 thin 253:0 167
docker-8:3-396359-pool: 0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing

Ping me if you need any other info :)

[rhvgoyal]

Thin pool seems to be ok. Check journalctl logs and see if something is there which gives details about why task failed.

If that does not, then enable libdm logging and re-run and see if something shows up. We need to know the details of why it failed.

Recently a daemon option was added to enable libdm logging. I think --storage-opt dm.libdm_log_level.

[liusdu]

Hi Guys, I got this problem. I found that when thin device is deleted but metadata of the layer exists. Then we can not remove this container, with error "Error response from daemon: devicemapper failed to remove the rootFS for : devicemapper : Error running DeleteDevice dm_task_run failed"

In the container-deleteing process, we delete the thin device first and then delete the metadata. So if docker is killing between these two actions. we can reproduce this issue.

[liusdu]

ping @rhvgoyal

[fencekicker]

I stumbled upon this on CentOS 7, using docker-1.13.1-162.git64e9980. In my case, our app is trying to restart a container, but the container was dead, and it could not be removed:

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
84c1b961192b xxxxxxxxxxxxxxxxx "/bin/sh -c '/usr/..." 3 weeks ago Dead xxxxxxxxxxxxxxxxx

I was seeing errors when trying to remove the container with docker rm -f:

# docker rm -fv xxxxxxx
Error response from daemon: Driver devicemapper failed to remove root filesystem 84c1b961192bae09505cce0e5b91f35dd1e44b0ae2f9375e9a17933ef088c16d: failed to remove device 3a363e90e5b2ac3601b9c4e758a17dd4ce7e260d3fc24c195a9a3841494acb0d-init:devicemapper: Error running DeleteDevice dm_task_run failed

I noticed @liusdu 's comment about the device getting deleted before metadata, so I thought maybe deleting the metadata would work. So I stopped the docker service, then:
# mv /var/lib/docker/devicemapper/metadata/3a363e90e5b2ac3601b9c4e758a17dd4ce7e260d3fc24c195a9a3841494acb0d-init .

(rm would be just as good, but I didn't expect this to work)
And then restarted docker, and what do you know:

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

To people who are stuck on CentOS 7 (we are, at least for the time being), I guess this can be used as a workaround.

[thaJeztah]

docker-1.13.1 is really old (and reached EOL), but I suspect you're running the Red Hat fork of Docker.

Note that CentOS 7 is still supported, and current versions of Docker for CentOS support overlay2 on CentOS (https://docs.docker.com/engine/install/centos/). Using the overlay2 storage driver may be a good alternative (but only a single storage driver can be used at a time, and switching storage drivers will make content stored with the other storage driver inaccessible (image in your image cache, and containers), so generally recommended to use a "fresh" install.

[fencekicker]

Yes, we're running the version shipped by RedHat in their extras repo. So you're saying we can also run the current Docker on CentOS 7? I hadn't thought of that, it's probably something worth trying.

[thaJeztah]

Running the Docker Engine should definitely work (but as mentioned, you may need to do a fresh install). That said, if you need to run other container tools from Red Hat's ecosystem, things can sometimes be somewhat complicated due to package dependencies (both the "docker" packages and red hat forked packages providing the same in the RPM metadata).