New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker 1.4 and "Driver aufs failed to remove root filesystem", "device or resource busy" #9665

Closed
jamshid opened this Issue Dec 14, 2014 · 82 comments

Comments

Projects
None yet
@jamshid
Contributor

jamshid commented Dec 14, 2014

I don't think I saw this under docker 1.3.x on aufs (boot2docker on OSX), though I have with devicemapper on centos server. Got this error after upgrading docker server, retrying the fig up worked.

I use brew and my docker client is still 1.3.2. Just saw the warning in the log below, I guess that's not really supported? I'll try upgrading docker client manually.

$ docker version
Client version: 1.3.2
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 39fa2fa
OS/Arch (client): darwin/amd64
Server version: 1.4.0
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 4595d4f
$ fig -f dns-fig.yml -p dns up -d
Recreating dns_hardfile_1...
Recreating dns_dnsmasq_1...
Cannot destroy container 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: Driver aufs failed to remove root filesystem 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: rename /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727 /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727-removing: device or resource busy
time="2014-12-14T20:21:27Z" level="info" msg="Container 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727 failed to exit within 10 seconds of SIGTERM - using the force" 
time="2014-12-14T20:21:27Z" level="debug" msg="Sending 9 to 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727" 
time="2014-12-14T20:21:28Z" level="info" msg="+job log(die, 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727, dns_dnsmasq:latest)" 
time="2014-12-14T20:21:28Z" level="info" msg="-job log(die, 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727, dns_dnsmasq:latest) = OK (0)" 
time="2014-12-14T20:21:28Z" level="info" msg="+job release_interface(07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727)"
...
time="2014-12-14T20:21:29Z" level="debug" msg="Calling DELETE /containers/{name:.*}" 
time="2014-12-14T20:21:29Z" level="info" msg="DELETE /v1.12/containers/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727?link=False&force=False&v=False" 
time="2014-12-14T20:21:29Z" level="info" msg="+job rm(07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727)" 
Cannot destroy container 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: Driver aufs failed to remove root filesystem 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: r
ename /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727 /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727
-removing: device or resource busy
time="2014-12-14T20:21:29Z" level="info" msg="-job rm(07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727) = ERR (1)" 
time="2014-12-14T20:21:29Z" level="error" msg="Handler for DELETE /containers/{name:.*} returned error: Cannot destroy container 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: Driver a
ufs failed to remove root filesystem 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: rename /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9
dd727 /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727-removing: device or resource busy" 
time="2014-12-14T20:21:29Z" level="error" msg="HTTP Error: statusCode=500 Cannot destroy container 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: Driver aufs failed to remove root file
system 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: rename /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727 /mnt/sda1/var/lib/docker
/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727-removing: device or resource busy" 
time="2014-12-14T20:24:15Z" level="debug" msg="Calling POST /containers/{name:.*}/exec" 
time="2014-12-14T20:24:15Z" level="info" msg="POST /v1.15/containers/dns_dnsmasq_1/exec" 
time="2014-12-14T20:24:15Z" level="debug" msg="Warning: client and server don't have the same version (client: 1.3.2, server: 1.4.0)"
@jamshid

This comment has been minimized.

Contributor

jamshid commented Dec 20, 2014

Still seeing this frequently on fig up -d, even at docker 1.4.1 on both client and server. Maybe having luck avoiding it with --no-recreate

FATA[0167] Error response from daemon: Cannot destroy container 74555d1513c4a9f3fc90ec0bb94bcf5b5ef558b6a54c5ff8a209ddd079fbf8f0: Driver aufs failed to remove root filesystem 74555d1513c4a9f3fc90ec0bb94bcf5b5ef558b6a54c5ff8a209ddd079fbf8f0: rename /mnt/sda1/var/lib/docker/aufs/mnt/74555d1513c4a9f3fc90ec0bb94bcf5b5ef558b6a54c5ff8a209ddd079fbf8f0 /mnt/sda1/var/lib/docker/aufs/mnt/74555d1513c4a9f3fc90ec0bb94bcf5b5ef558b6a54c5ff8a209ddd079fbf8f0-removing: device or resource busy 
$ docker info
Containers: 28
Images: 1356
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Dirs: 1424
Execution Driver: native-0.2
Kernel Version: 3.16.7-tinycore64
Operating System: Boot2Docker 1.4.1 (TCL 5.4); master : 86f7ec8 - Tue Dec 16 23:11:29 UTC 2014
CPUs: 8
Total Memory: 3.866 GiB
Name: boot2docker
ID: 665E:OA4X:64MF:PVJF:WKXR:Q7V2:455C:HS5K:SYQW:VZIU:SI4C:HL5M
Debug mode (server): true
Debug mode (client): false
Fds: 110
Goroutines: 83
EventsListeners: 1
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
@OscarBarrett

This comment has been minimized.

OscarBarrett commented Jan 4, 2015

I'm seeing the same issue on 1.4.1.

$ docker info
Containers: 1
Images: 11
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Dirs: 17
Execution Driver: native-0.2
Kernel Version: 3.16.7-tinycore64
Operating System: Boot2Docker 1.4.1 (TCL 5.4); master : 86f7ec8 - Tue Dec 16 23:11:29 UTC 2014
CPUs: 8
Total Memory: 1.961 GiB
Name: boot2docker
ID: LBNP:H3GO:XVDK:GTCP:FNOX:OPY4:QJUP:QIKK:HAKK:TABZ:3NP6:4N7S
Debug mode (server): true
Debug mode (client): false
Fds: 17
Goroutines: 18
EventsListeners: 0
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
@skozin

This comment has been minimized.

skozin commented Jan 20, 2015

I'm seeing this too on Docker 1.4.1 (both client and server). But target container seems to be destroyed, at least it is not present in the docker ps -a output.

This may be unrelated, but just in case: I perform periodic health checks using ephemeral containers attached to the target container's network (via the --net container:xyz flag). It seems that one of these containers was started right before the moment target container died. As a result, this check container failed to start successfully because target network was already destroyed (and this is totally ok).

Here is the log (I replaced ids of target and check containers with TARGET and CHECK for clarity):

//
// Creating and starting target container
// docker run -it --rm -p 10.0.42.1:4000:3000 test-app
//

time="2015-01-20T19:36:17Z" level="info" msg="POST /v1.16/containers/create" 
time="2015-01-20T19:36:17Z" level="info" msg="+job create()" 
time="2015-01-20T19:36:17Z" level="info" msg="+job log(create, TARGET, test-app:latest)" 
time="2015-01-20T19:36:17Z" level="info" msg="-job log(create, TARGET, test-app:latest) = OK (0)" 
time="2015-01-20T19:36:17Z" level="info" msg="-job create() = OK (0)" 
time="2015-01-20T19:36:17Z" level="info" msg="POST /v1.16/containers/TARGET/attach?stderr=1&stdin=1&stdout=1&stream=1" 
time="2015-01-20T19:36:17Z" level="info" msg="+job container_inspect(TARGET)" 
time="2015-01-20T19:36:17Z" level="info" msg="-job container_inspect(TARGET) = OK (0)" 
time="2015-01-20T19:36:17Z" level="info" msg="+job attach(TARGET)" 
time="2015-01-20T19:36:17Z" level="info" msg="POST /v1.16/containers/TARGET/start" 
time="2015-01-20T19:36:17Z" level="info" msg="+job start(TARGET)" 
time="2015-01-20T19:36:17Z" level="info" msg="+job allocate_interface(TARGET)" 
time="2015-01-20T19:36:17Z" level="info" msg="-job allocate_interface(TARGET) = OK (0)" 
time="2015-01-20T19:36:17Z" level="info" msg="+job allocate_port(TARGET)" 
time="2015-01-20T19:36:17Z" level="info" msg="-job allocate_port(TARGET) = OK (0)" 
time="2015-01-20T19:36:17Z" level="info" msg="+job log(start, TARGET, test-app:latest)" 
time="2015-01-20T19:36:17Z" level="info" msg="-job log(start, TARGET, test-app:latest) = OK (0)" 
time="2015-01-20T19:36:17Z" level="info" msg="GET /containers/TARGET/json" 
time="2015-01-20T19:36:17Z" level="info" msg="+job container_inspect(TARGET)" 
time="2015-01-20T19:36:17Z" level="info" msg="-job start(TARGET) = OK (0)" 
time="2015-01-20T19:36:17Z" level="info" msg="-job container_inspect(TARGET) = OK (0)" 
time="2015-01-20T19:36:17Z" level="info" msg="POST /v1.16/containers/TARGET/resize?h=25&w=180" 
time="2015-01-20T19:36:17Z" level="info" msg="+job resize(TARGET, 25, 180)" 
time="2015-01-20T19:36:17Z" level="info" msg="-job resize(TARGET, 25, 180) = OK (0)"

// ... (unrelated log skipped)
//
// Creating and starting ephemeral check container
// docker run --name chk-N --net container:TARGET check-img some_check_command
//

time="2015-01-20T19:36:46Z" level="info" msg="POST /v1.16/containers/create?name=chk-N" 
time="2015-01-20T19:36:46Z" level="info" msg="+job create(chk-N)" 
time="2015-01-20T19:36:46Z" level="info" msg="+job log(create, CHECK, consul:latest)" 
time="2015-01-20T19:36:46Z" level="info" msg="-job log(create, CHECK, consul:latest) = OK (0)" 
time="2015-01-20T19:36:46Z" level="info" msg="-job create(chk-N) = OK (0)" 
time="2015-01-20T19:36:46Z" level="info" msg="POST /v1.16/containers/CHECK/attach?stderr=1&stdout=1&stream=1" 
time="2015-01-20T19:36:46Z" level="info" msg="+job container_inspect(CHECK)" 
time="2015-01-20T19:36:46Z" level="info" msg="-job container_inspect(CHECK) = OK (0)" 
time="2015-01-20T19:36:46Z" level="info" msg="+job attach(CHECK)" 
time="2015-01-20T19:36:46Z" level="info" msg="POST /v1.16/containers/CHECK/start" 
time="2015-01-20T19:36:46Z" level="info" msg="+job start(CHECK)" 
time="2015-01-20T19:36:46Z" level="info" msg="+job log(start, CHECK, consul:latest)" 
time="2015-01-20T19:36:46Z" level="info" msg="-job log(start, CHECK, consul:latest) = OK (0)"

//
// Target container died
//

time="2015-01-20T19:36:47Z" level="info" msg="+job log(die, TARGET, test-app:latest)" 
time="2015-01-20T19:36:47Z" level="info" msg="-job log(die, TARGET, test-app:latest) = OK (0)" 
time="2015-01-20T19:36:47Z" level="info" msg="+job release_interface(TARGET)"

time="2015-01-20T19:36:47Z" level="info" msg="GET /containers/CHECK/json" 
time="2015-01-20T19:36:47Z" level="info" msg="+job container_inspect(CHECK)"

time="2015-01-20T19:36:47Z" level="info" msg="-job attach(TARGET) = OK (0)" 
time="2015-01-20T19:36:47Z" level="info" msg="POST /v1.16/containers/TARGET/wait" 
time="2015-01-20T19:36:47Z" level="info" msg="+job wait(TARGET)" 
time="2015-01-20T19:36:47Z" level="info" msg="-job release_interface(TARGET) = OK (0)" 
time="2015-01-20T19:36:47Z" level="info" msg="-job wait(TARGET) = OK (0)" 
time="2015-01-20T19:36:47Z" level="info" msg="GET /v1.16/containers/TARGET/json" 
time="2015-01-20T19:36:47Z" level="info" msg="+job container_inspect(TARGET)" 
time="2015-01-20T19:36:47Z" level="info" msg="-job container_inspect(TARGET) = OK (0)" 

//
// Removing target container (as a consequence of --rm cli flag)
//

time="2015-01-20T19:36:47Z" level="info" msg="DELETE /v1.16/containers/TARGET?v=1" 
time="2015-01-20T19:36:47Z" level="info" msg="+job rm(TARGET)" 

//
// Error removing target container: device or resource busy. This is not OK.
//

Cannot destroy container TARGET: Driver aufs failed to remove root filesystem TARGET: rename /mnt/sda1/var/lib/docker/aufs/mnt/TARGET /mnt/sda1/var/lib/docker/aufs/mnt/TARGET-removing: device or resource busy
time="2015-01-20T19:36:47Z" level="info" msg="-job rm(TARGET) = ERR (1)" 
time="2015-01-20T19:36:47Z" level="error" msg="Handler for DELETE /containers/{name:.*} returned error: Cannot destroy container TARGET: Driver aufs failed to remove root filesystem TARGET: rename /mnt/sda1/var/lib/docker/aufs/mnt/TARGET /mnt/sda1/var/lib/docker/aufs/mnt/TARGET-removing: device or resource busy" 
time="2015-01-20T19:36:47Z" level="error" msg="HTTP Error: statusCode=500 Cannot destroy container TARGET: Driver aufs failed to remove root filesystem TARGET: rename /mnt/sda1/var/lib/docker/aufs/mnt/TARGET /mnt/sda1/var/lib/docker/aufs/mnt/TARGET-removing: device or resource busy" 

//
// Error starting check container: TARGET network is already destroyed. This is OK.
//

time="2015-01-20T19:36:47Z" level="info" msg="+job release_interface(CHECK)" 
No network information to release for CHECK
time="2015-01-20T19:36:47Z" level="info" msg="-job release_interface(CHECK) = ERR (1)" 
time="2015-01-20T19:36:47Z" level="info" msg="-job attach(CHECK) = OK (0)" 
time="2015-01-20T19:36:47Z" level="info" msg="+job release_interface(CHECK)" 
No network information to release for CHECK
time="2015-01-20T19:36:47Z" level="info" msg="-job release_interface(CHECK) = ERR (1)" 
time="2015-01-20T19:36:47Z" level="info" msg="+job log(die, CHECK, consul:latest)" 
time="2015-01-20T19:36:47Z" level="info" msg="-job log(die, CHECK, consul:latest) = OK (0)" 
Cannot start container CHECK: setup networking failed get network namespace fd: open /proc/10959/ns/net: no such file or directory
time="2015-01-20T19:36:47Z" level="info" msg="-job start(CHECK) = ERR (1)" 
time="2015-01-20T19:36:47Z" level="error" msg="Handler for POST /containers/{name:.*}/start returned error: Cannot start container CHECK: setup networking failed get network namespace fd: open /proc/10959/ns/net: no such file or directory" 
time="2015-01-20T19:36:47Z" level="error" msg="HTTP Error: statusCode=404 Cannot start container CHECK: setup networking failed get network namespace fd: open /proc/10959/ns/net: no such file or directory" 

Docker version & info:

$ docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): darwin/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

$ docker info
Containers: 5
Images: 105
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Dirs: 117
Execution Driver: native-0.2
Kernel Version: 3.16.7-tinycore64
Operating System: Boot2Docker 1.4.1 (TCL 5.4); master : 86f7ec8 - Tue Dec 16 23:11:29 UTC 2014
CPUs: 4
Total Memory: 1.961 GiB
Name: boot2docker
ID: LP33:7HWL:DFI6:5AL5:RNBL:D5ZJ:AMGD:WO3I:DGTW:SKHZ:3WFW:KNMJ
Debug mode (server): true
Debug mode (client): false
Fds: 36
Goroutines: 55
EventsListeners: 1
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
@remmelt

This comment has been minimized.

remmelt commented Feb 2, 2015

Same here, although not through boot2docker but on Ubuntu 14.04 directly. We have a bunch of health checks through progrium's docker-consul which are just curl commands that run inside a container. These containers are frequently not removed. They are created using a bash script that creates a container from within the consul container by having the docker.sock mounted. It's using docker run --rm --net container:$container_id ..... as well.
The event stream shows that most health checking containers do die and destroy correctly.

% docker info
Containers: 3596
Images: 222
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 7423
Execution Driver: native-0.2
Kernel Version: 3.13.0-44-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 4
Total Memory: 15.67 GiB
Name: docker02
ID: EOVE:DWGJ:AE2W:YLT5:F27M:IEJL:Q33E:FQCJ:ASIN:EHCM:ACH4:WJND
Username: bvadevops
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support
% docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8
@skozin

This comment has been minimized.

skozin commented Feb 2, 2015

@remmelt, these containers are not removed because Docker ignores --rm flag when there is a problem launching container. This is a weird and probably undocumented, but expected behavior of Docker, so what you're seeing is in fact a docker-consul's issue.

See this comment for more info. I made a fork that resolves this and several other issues of docker-consul. Here is the related commit. I'll try to get these fixes merged back to the original repo, but probably later, as I have really little time now.

The issue being discussed here occurs to me when I try to remove target container (i.e. the container that health checks run against). And it is accompanied by Driver aufs failed to remove root filesystem error message. And, despite of an error message, the container gets removed, at least it disappears from the docker ps -a list.

@remmelt

This comment has been minimized.

remmelt commented Feb 3, 2015

Thank you!

@exarkun

This comment has been minimized.

Contributor

exarkun commented Feb 3, 2015

I also see this error sometimes when running the Flocker test suite. For example:

[ERROR]
Traceback (most recent call last):
  File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/trial/util.py", line 296, in _runSequentially
    thing = yield d
  File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 196, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/home/exarkun/Projects/flocker/master/flocker/node/_docker.py", line 507, in _remove
    self._client.remove_container(container_name)
  File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/docker/client.py", line 896, in remove_container
    self._raise_for_status(res)
  File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/docker/client.py", line 94, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("Cannot destroy container DockerClientTests-flocker-node-functional-test_docker-DockerClientTests-test_list_removed_containers-888568622283900587922261: Driver aufs failed to remove root filesystem 83692922f272e460b71efdecf129064d3279f5072c2c1e70afcb38c73798c2fa: rename /var/lib/docker/aufs/diff/83692922f272e460b71efdecf129064d3279f5072c2c1e70afcb38c73798c2fa /var/lib/docker/aufs/diff/83692922f272e460b71efdecf129064d3279f5072c2c1e70afcb38c73798c2fa-removing: device or resource busy")

flocker.node.functional.test_docker.DockerClientTests.test_list_removed_containers

and

[ERROR]
Traceback (most recent call last):
  File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 196, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/home/exarkun/Projects/flocker/master/flocker/node/_docker.py", line 507, in _remove
    self._client.remove_container(container_name)
  File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/docker/client.py", line 896, in remove_container
    self._raise_for_status(res)
  File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/docker/client.py", line 94, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("Cannot destroy container flocker--912891792647--622134077079: Driver aufs failed to remove root filesystem 005d96d8ce4ca2d74242cc3873772274dff60fb5980b2ce05413d18e9d0d9d70: rename /var/lib/docker/aufs/diff/005d96d8ce4ca2d74242cc3873772274dff60fb5980b2ce05413d18e9d0d9d70 /var/lib/docker/aufs/diff/005d96d8ce4ca2d74242cc3873772274dff60fb5980b2ce05413d18e9d0d9d70-removing: device or resource busy")

flocker.node.functional.test_docker.IDockerClientNamespacedTests.test_add_and_remove

These are both tests that use the Docker API to create and then quickly stop and remove a container.

$ sudo docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8
$ sudo docker info
Containers: 3
Images: 39
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 51
Execution Driver: native-0.2
Kernel Version: 3.13.0-44-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 8
Total Memory: 62.89 GiB
Name: preon
ID: REGQ:VSJY:T7LE:7RHM:GCW7:QHGZ:K2YJ:L72Z:KLP3:54DY:JO27:EM35
WARNING: No swap limit support
@nomeata

This comment has been minimized.

nomeata commented Mar 5, 2015

I’m also observing this, docker 1.4.1 on Linux 3.16.0-0.bpo.4-amd64.

@nomeata

This comment has been minimized.

nomeata commented Mar 5, 2015

And some numbers: Of 1820 created short-living containers, 240 caused this error.

This is particularly annoying as docker run will print this error message on stderr, confusing tools that use the output of that to determine whether the actual job was successful.

@nomeata

This comment has been minimized.

nomeata commented Mar 5, 2015

Upgraded to 1.5.0, same problem.

@g-ovidiupl

This comment has been minimized.

g-ovidiupl commented Apr 2, 2015

Simple repro case. Under zsh, run:

while true; do echo "exit" | docker run -i --rm busybox; if (( ${?} != 0 )); then break; fi; done

Run 2-3 parallel instances of the script, if desired.

Environment:

~% docker info % 15-04-02 15:07
Containers: 222
Images: 4
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 450
Execution Driver: native-0.2
Kernel Version: 3.13.0-48-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 12
Total Memory: 23.54 GiB
Name: REDACTED
ID: REDACTED
Debug mode (server): true
Debug mode (client): false
Fds: 18
Goroutines: 26
EventsListeners: 0
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker
WARNING: No swap limit support
~% docker version % 15-04-02 15:07
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
~% % 15-04-02 15:07

@exarkun

This comment has been minimized.

Contributor

exarkun commented Apr 3, 2015

I tried the reproducer but unfortunately wasn't able to trigger this failure. How long do you typically run this before you see a problem?

Here's my docker info:

Containers: 20
Images: 231
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 271
Execution Driver: native-0.2
Kernel Version: 3.13.0-44-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 8
Total Memory: 62.89 GiB
Name: ...
ID: ...
WARNING: No swap limit support

and docker version

Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
@g-ovidiupl

This comment has been minimized.

g-ovidiupl commented Apr 3, 2015

At home, I can repro this in a few seconds on an Ubuntu instance under VMWare (on a Windows host). I don't have the docker info/version handy at this time.

At work, I can repro this in at most 15-20 minutes with 2 scripts running concurrently (on the configuration described above). I'll try more concurrent scripts today.

@ckulla

This comment has been minimized.

ckulla commented Apr 7, 2015

I experience this problem in ubuntu-14.04 on a vmware host

@fishtek

This comment has been minimized.

fishtek commented Apr 9, 2015

+1

Also with the Docker v1.6.0 RC5 on Ubuntu 12.04 with trusty backport kernel:

sysadmin@slave2:~$ sudo docker info && sudo docker version
Containers: 1
Images: 805
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 807
 Dirperm1 Supported: false
Execution Driver: native-0.2
Kernel Version: 3.13.0-49-generic
Operating System: Ubuntu precise (12.04.5 LTS)
CPUs: 4
Total Memory: 3.86 GiB
Name: slave2
ID: L2Y4:4XL5:J56E:SWOQ:EV7G:FBTU:4ACZ:4LBB:RAQW:FBNP:MT32:7JH7
Client version: 1.6.0-rc5
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): fc4825d
OS/Arch (client): linux/amd64
Server version: 1.6.0-rc5
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): fc4825d
OS/Arch (server): linux/amd64

And with the current Docker v1.5.0 on Ubuntu 12.04 with trusty backport kernel:

sysadmin@slave2:~$ sudo docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
sysadmin@slave2:~$ 
sysadmin@slave2:~$ sudo docker info
Containers: 118
Images: 1510
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1888
Execution Driver: native-0.2
Kernel Version: 3.13.0-48-generic
Operating System: Ubuntu precise (12.04.5 LTS)
CPUs: 4
Total Memory: 3.86 GiB
Name: slave2
ID: L2Y4:4XL5:J56E:SWOQ:EV7G:FBTU:4ACZ:4LBB:RAQW:FBNP:MT32:7JH7
@oncletom

This comment has been minimized.

oncletom commented Apr 13, 2015

I encounter this issue as well with docker 1.4.1. It happens on a busy machine within a processing pipeline:

Error response from daemon: Cannot destroy container eccd3089dc18912001f2047431810e14c4b1808b470bac063a39f87bee16d5be:
  Driver aufs failed to remove root filesystem eccd3089dc18912001f2047431810e14c4b1808b470bac063a39f87bee16d5be: rename /var/lib/docker/aufs/diff/eccd3089dc18912001f2047431810e14c4b1808b470bac063a39f87bee16d5be
/var/lib/docker/aufs/diff/eccd3089dc18912001f2047431810e14c4b1808b470bac063a39f87bee16d5be-removing: device or resource busy
@TalonOne

This comment has been minimized.

TalonOne commented Apr 17, 2015

+1

I am seeing the same as well with docker 1.6.0. I dont really have anything complicated running on the machine, but the containers I am trying to remove seem to be dead, if that matters.

sysadmin@docker10:~$ tail -F /var/logs/docker.log
time="2015-04-17T12:25:31Z" level=info msg="DELETE /v1.18/containers/b49bb2f86b12" 
time="2015-04-17T12:25:31Z" level=info msg="+job rm(b49bb2f86b12)" 
Cannot destroy container b49bb2f86b12: Driver aufs failed to remove root filesystem b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb: rename /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb-removing: device or resource busy
time="2015-04-17T12:25:31Z" level=info msg="-job rm(b49bb2f86b12) = ERR (1)" 
time="2015-04-17T12:25:31Z" level=error msg="Handler for DELETE /containers/{name:.*} returned error: Cannot destroy container b49bb2f86b12: Driver aufs failed to remove root filesystem b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb: rename /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb-removing: device or resource busy" 
time="2015-04-17T12:25:31Z" level=error msg="HTTP Error: statusCode=500 Cannot destroy container b49bb2f86b12: Driver aufs failed to remove root filesystem b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb: rename /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb-removing: device or resource busy" 
Error response from daemon: Cannot destroy container b49bb2f86b12: Driver aufs failed to remove root filesystem b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb: rename /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb-removing: device or resource busy
time="2015-04-17T12:25:31Z" level=info msg="DELETE /v1.18/containers/eac104ae9157" 
time="2015-04-17T12:25:31Z" level=info msg="+job rm(eac104ae9157)" 
Cannot destroy container eac104ae9157: Driver aufs failed to remove root filesystem eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8: rename /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8 /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8-removing: device or resource busy
time="2015-04-17T12:25:31Z" level=info msg="-job rm(eac104ae9157) = ERR (1)" 
time="2015-04-17T12:25:31Z" level=error msg="Handler for DELETE /containers/{name:.*} returned error: Cannot destroy container eac104ae9157: Driver aufs failed to remove root filesystem eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8: rename /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8 /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8-removing: device or resource busy" 
time="2015-04-17T12:25:31Z" level=error msg="HTTP Error: statusCode=500 Cannot destroy container eac104ae9157: Driver aufs failed to remove root filesystem eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8: rename /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8 /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8-removing: device or resource busy" 
Error response from daemon: Cannot destroy container eac104ae9157: Driver aufs failed to remove root filesystem eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8: rename /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8 /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8-removing: device or resource busy
sysadmin@docker10:~$ sudo docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64
sysadmin@docker10:~$ sudo docker info
Containers: 3
Images: 81
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 87
 Dirperm1 Supported: false
Execution Driver: native-0.2
Kernel Version: 3.10.48-3.10.6-amd64-13160847
Operating System: Ubuntu precise (12.04.2 LTS)
CPUs: 8
Total Memory: 15.68 GiB
Name: docker10
ID: CR2N:5JNE:S6XA:NNXS:FEQD:IVTM:7NCL:INCB:3AAS:CCKJ:XWUU:5FF2
sysadmin@docker10:~$ sudo docker ps -a
CONTAINER ID        IMAGE                                                              COMMAND                CREATED             STATUS              PORTS                     NAMES
8aa56e3ba83f        gliderlabs/registrator:latest                                      "/bin/sh -c '/bin/re   About an hour ago   Up About an hour                              dope_container   
b49bb2f86b12        hub.abc.com/application/test-img:latest                            "/bin/sh -c '/usr/lo   20 hours ago        Dead                0.0.0.0:31000->8080/tcp                                                
eac104ae9157        gliderlabs/registrator:latest                                      "/bin/sh -c '/bin/re   20 hours ago        Dead                                                                                       
@Solutron

This comment has been minimized.

Solutron commented Apr 17, 2015

Albeit I'm on an older version of Ubuntu:

13:13:10-ubuntu~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 13.10
Release:    13.10
Codename:   saucy

My guess is that I'd see this in newer versions as well based on the thread here.
I observed this error today as well:

13:05:17-ubuntu~$ sudo service docker stop
docker stop/waiting
13:06:26-ubuntu~$ sudo rm -rf /var/lib/docker
rm: cannot remove ‘/var/lib/docker/aufs’: Device or resource busy
13:06:29-ubuntu~$ ps aux | grep dockerubuntu    4313  0.0  0.0   9692   948 pts/9    S+   13:06   0:00 grep --color=auto docker
13:06:47-ubuntu~$ 
@dnephin

This comment has been minimized.

Member

dnephin commented Apr 22, 2015

I hit this on docker version: 1.6.0.

I was able to remove the container with docker rm -f

@jamshid

This comment has been minimized.

Contributor

jamshid commented Apr 24, 2015

I originally reported this in Dec 2014 with docker 1.3.x. FWIW I've been using docker-compose and docker client&server 1.5 and 1.6 with boot2docker on osx and docker-machine, and have not seen the Cannot destroy container error.

@Freedomkk

This comment has been minimized.

Freedomkk commented Apr 24, 2015

I hit this on docker version: 1.6.0
the container cannot stop,just hang on it
and after restart docker service
it cannot remove

@billsmith

This comment has been minimized.

billsmith commented May 8, 2015

I saw this on docker 1.6.0 build 4749651 on Ubuntu 14.04. I had some containers (not sure whether they were running on killed) on docker 1.5.0. I upgraded to docker 1.6.0 and then immediately afterwards tried to remove the containers.

docker rm -f seemed to work

@jamshid

This comment has been minimized.

Contributor

jamshid commented May 9, 2015

I spoke too soon, just saw this while doing a few simultaneous Dockerfile builds with docker 1.6.0 client on OSX against a digitalocean 1.6.1 server.

Error removing intermediate container a0568433a48a: Driver aufs failed to remove root filesystem a0568433a4...: rename /var/lib/docker/aufs/diff/a0568433a48a... /var/lib/docker/aufs/diff/a0568433a4...-removing: device or resource busy
@rakeshcusat

This comment has been minimized.

rakeshcusat commented Nov 22, 2015

We were also noticing similar issue Driver aufs failed to remove root filesystem, but after upgrading kernel to 3.19.0-25-generic we don't see this error anymore.

@garyritchie

This comment has been minimized.

garyritchie commented Dec 1, 2015

@rmetzler 's explanation helped me out: docker stop cadvisor; then docker rm [dead container]

@zfil

This comment has been minimized.

zfil commented Dec 17, 2015

I got this error with kernel 3.19.0-39 too see #3968 (comment)

@MrCoder

This comment has been minimized.

MrCoder commented Dec 29, 2015

Getting this error for Driver devicemapper.

webdev2:~ # docker version
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:
 OS/Arch:      linux/amd64
webdev2:~ # docker ps -a
CONTAINER ID        IMAGE                       COMMAND             CREATED             STATUS              PORTS               NAMES
6efab667b88d        openshift/origin-pod:v1.1   "/pod"              7 days ago          Dead
webdev2:~ # docker rm 6ef
Error response from daemon: Cannot destroy container 6ef: Driver devicemapper failed to remove root filesystem 6efab667b88d8aa1ad4e4a89582f3d21a4f4f6fba2dc9df018eb711c34df1ed6: Device is Busy
Error: failed to remove containers: [6ef]
@ostretsov

This comment has been minimized.

ostretsov commented Dec 30, 2015

Same problem and solved it just with umount:

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
6563dd842a83        postgres:9.4        "/docker-entrypoint.s"   25 hours ago        Dead                                    tiny_franklin
ac749a08434d        postgres:9.4        "/docker-entrypoint.s"   25 hours ago        Dead                                    trusting_kirch

$ mount
...
none on /var/lib/docker/aufs/mnt/ac749a08434dd9508503c6bdd43dfb09726646a4a79593500d8f5c913cda3a71 type aufs (rw,relatime,si=66a2e49a9d580fd1,dio)
none on /var/lib/docker/aufs/mnt/6563dd842a838b42eac342d3aef4ba2f4a989dca8d9cd562725687e8e23a3d5c type aufs (rw,relatime,si=66a2e49e760e7fd1,dio)

$ sudo umount /var/lib/docker/aufs/mnt/ac749a08434dd9508503c6bdd43dfb09726646a4a79593500d8f5c913cda3a71
$ sudo umount /var/lib/docker/aufs/mnt/6563dd842a838b42eac342d3aef4ba2f4a989dca8d9cd562725687e8e23a3d5c

$ docker rm 6563
$ docker rm ac74
$ docker ps -a
(empty list)
@g-ovidiupl

This comment has been minimized.

g-ovidiupl commented Dec 30, 2015

@ruFog That misses the point. Of course "umount" solves the problem, if you wait long enough. The problem is that stopping a container does a "umount" internally, and that fails due to a race condition with some unknown thing that keeps file descriptors around for the device.

Sure, if you want a workaround, "until sudo umount ${MOUNT_POINT}; do sleep 5; done". But that's the problem to begin with, Docker (or possibly some lower-level culprit) should do that correctly by itself.

claytono pushed a commit to twc-openstack/garethr-docker that referenced this issue Jan 11, 2016

Clayton O'Neill
Work around docker rm errors
This is a hopefully temporary work around this issue:
moby/moby#9665

This means that docker rm is unreliable.  All we really care about when
we stop the service is to ensure that the service is stopped, so we
explicitly ask docker for that via inspect instead of relying on the rm
result code.

claytono pushed a commit to twc-openstack/garethr-docker that referenced this issue Jan 11, 2016

Clayton O'Neill
Work around docker rm errors
This is a hopefully temporary work around this issue:
moby/moby#9665

This means that docker rm is unreliable.  All we really care about when
we stop the service is to ensure that the service is stopped, so we
explicitly ask docker for that via inspect instead of relying on the rm
result code.
@tfoote

This comment has been minimized.

tfoote commented Jan 14, 2016

I'm seeing this error during a build stage in docker 1.9.1. It continued to function but demonstrated this error. full log

This machine would have been under relatively heavy load processing 4 parallel build + test processes in docker containers. Typical load average for these machines is ~2.

Here's the relevant console outputs.

17:06:42 Step 17 : ENV ROSDISTRO_INDEX_URL https://raw.githubusercontent.com/ros/rosdistro/master/index.yaml
17:06:42  ---> Running in 9e109f73e134
17:06:43  ---> 89f69407c123
17:06:43 Error removing intermediate container 9e109f73e134: rmdriverfs: Driver aufs failed to remove root filesystem 9e109f73e134805c928c70f9f393726d861a74f45d96e624230a954b5d44a67f: rename /var/lib/docker/aufs/mnt/9e109f73e134805c928c70f9f393726d861a74f45d96e624230a954b5d44a67f /var/lib/docker/aufs/mnt/9e109f73e134805c928c70f9f393726d861a74f45d96e624230a954b5d44a67f-removing: device or resource busy
17:06:43 Step 18 : RUN rosdep init

Docker version and docker info from the logs.

# BEGIN SECTION: docker version
17:04:47 + docker version
17:04:47 Client:
17:04:47  Version:      1.9.1
17:04:47  API version:  1.21
17:04:47  Go version:   go1.4.2
17:04:47  Git commit:   a34a1d5
17:04:47  Built:        Fri Nov 20 13:12:04 UTC 2015
17:04:47  OS/Arch:      linux/amd64
17:04:47 
17:04:47 Server:
17:04:47  Version:      1.9.1
17:04:47  API version:  1.21
17:04:47  Go version:   go1.4.2
17:04:47  Git commit:   a34a1d5
17:04:47  Built:        Fri Nov 20 13:12:04 UTC 2015
17:04:47  OS/Arch:      linux/amd64
17:04:47 + echo # END SECTION
17:04:47 # END SECTION
17:04:47 + echo # BEGIN SECTION: docker info
17:04:47 
3. docker info
Hide Details

# BEGIN SECTION: docker info
17:04:47 + docker info
17:04:51 Containers: 319
17:04:51 Images: 21908
17:04:51 Server Version: 1.9.1
17:04:51 Storage Driver: aufs
17:04:51  Root Dir: /var/lib/docker/aufs
17:04:51  Backing Filesystem: extfs
17:04:51  Dirs: 22924
17:04:51  Dirperm1 Supported: false
17:04:51 Execution Driver: native-0.2
17:04:51 Logging Driver: json-file
17:04:51 Kernel Version: 3.13.0-48-generic
17:04:51 Operating System: Ubuntu 14.04.2 LTS
17:04:51 CPUs: 4
17:04:51 Total Memory: 15.67 GiB
17:04:51 Name: ip-172-31-6-102
17:04:51 ID: F26F:MOG6:HDN5:LSVU:VHW7:TNNK:NBRA:JVQG:H3CH:ZDUD:PHMK:7YLU
17:04:51 WARNING: No swap limit support
17:04:51 + echo # END SECTION
17:04:51 # END SECTION
17:04:51 [Idoc__carl_estop__ubuntu_trusty_amd64] $ /bin/sh -xe /tmp/hudson8392598443058835758.sh
17:04:51 + echo # BEGIN SECTION: Check docker status

Unfortunately this instance has been rotated out before I was able to debug into these logs.

However searching through my logs I find 2016 instances of this error in about 25000 attempts.

A snippet can be seen here:

Jbin_uU64__sound_play__ubuntu_utopic_amd64__binary/builds/1/log:Error removing intermediate container e02bb533fc41: rmdriverfs: Driver aufs failed to remove root filesystem e02bb533fc418c3cc134d7f64791e335e023c8934aacb774d7cc39554bb57573: rename /var/lib/docker/aufs/mnt/e02bb533fc418c3cc134d7f64791e335e023c8934aacb774d7cc39554bb57573 /var/lib/docker/aufs/mnt/e02bb533fc418c3cc134d7f64791e335e023c8934aacb774d7cc39554bb57573-removing: device or resource busy
Jbin_uU64__ueye__ubuntu_utopic_amd64__binary/builds/1/log:Error removing intermediate container c44f099d1bff: rmdriverfs: Driver aufs failed to remove root filesystem c44f099d1bffc73efa7e51a5549f547d20575dde21ab0e38dc87bd05c9184e05: rename /var/lib/docker/aufs/diff/c44f099d1bffc73efa7e51a5549f547d20575dde21ab0e38dc87bd05c9184e05 /var/lib/docker/aufs/diff/c44f099d1bffc73efa7e51a5549f547d20575dde21ab0e38dc87bd05c9184e05-removing: device or resource busy
Jbin_uU32__marti_sensor_msgs__ubuntu_utopic_i386__binary/builds/1/log:Error removing intermediate container 6c9addccada3: rmdriverfs: Driver aufs failed to remove root filesystem 6c9addccada3d3613727b7563db43c3320b503f28a742116f5d4d0b289f01c22: rename /var/lib/docker/aufs/mnt/6c9addccada3d3613727b7563db43c3320b503f28a742116f5d4d0b289f01c22 /var/lib/docker/aufs/mnt/6c9addccada3d3613727b7563db43c3320b503f28a742116f5d4d0b289f01c22-removing: device or resource busy
Jbin_uU64__rosapi__ubuntu_utopic_amd64__binary/builds/1/log:Error removing intermediate container d48e636a6d04: rmdriverfs: Driver aufs failed to remove root filesystem d48e636a6d042428382e7a2456ba3a4c5031d06db18a0556a664b21fbbff2747: rename /var/lib/docker/aufs/mnt/d48e636a6d042428382e7a2456ba3a4c5031d06db18a0556a664b21fbbff2747 /var/lib/docker/aufs/mnt/d48e636a6d042428382e7a2456ba3a4c5031d06db18a0556a664b21fbbff2747-removing: device or resource busy
Jbin_uT64__collada_parser__ubuntu_trusty_amd64__binary/builds/1/log:Error removing intermediate container d30a9fa5935c: rmdriverfs: Driver aufs failed to remove root filesystem d30a9fa5935c862320dffc95fe2f4a0dfb98ed1417b70d180445c4e346e1743b: rename /var/lib/docker/aufs/mnt/d30a9fa5935c862320dffc95fe2f4a0dfb98ed1417b70d180445c4e346e1743b /var/lib/docker/aufs/mnt/d30a9fa5935c862320dffc95fe2f4a0dfb98ed1417b70d180445c4e346e1743b-removing: device or resource busy
Jsrc_uV__shape_msgs__ubuntu_vivid__source/builds/1/log:Error removing intermediate container 581391a82307: rmdriverfs: Driver aufs failed to remove root filesystem 581391a82307806d5ab3e5bb5c4ad22ad7364541bc24f5353fbf05d025c0fd59: rename /var/lib/docker/aufs/mnt/581391a82307806d5ab3e5bb5c4ad22ad7364541bc24f5353fbf05d025c0fd59 /var/lib/docker/aufs/mnt/581391a82307806d5ab3e5bb5c4ad22ad7364541bc24f5353fbf05d025c0fd59-removing: device or resource busy
Jbin_uU64__grid_map__ubuntu_utopic_amd64__binary/builds/2/log:Error removing intermediate container b383b64333a6: rmdriverfs: Driver aufs failed to remove root filesystem b383b64333a6973088a25c3b3b40e50d8d92e6c1c3e6f126c3625e14d5240c7a: rename /var/lib/docker/aufs/mnt/b383b64333a6973088a25c3b3b40e50d8d92e6c1c3e6f126c3625e14d5240c7a /var/lib/docker/aufs/mnt/b383b64333a6973088a25c3b3b40e50d8d92e6c1c3e6f126c3625e14d5240c7a-removing: device or resource busy
Ibin_arm_uThf__mrp2_hardware__ubuntu_trusty_armhf__binary/builds/1/log:Error removing intermediate container 86bda4cf14aa: rmdriverfs: Driver aufs failed to remove root filesystem 86bda4cf14aa2a7c7bb3fe2999e61951c11df59522f31467d287277f8c8b874b: rename /var/lib/docker/aufs/diff/86bda4cf14aa2a7c7bb3fe2999e61951c11df59522f31467d287277f8c8b874b /var/lib/docker/aufs/diff/86bda4cf14aa2a7c7bb3fe2999e61951c11df59522f31467d287277f8c8b874b-removing: device or resource busy
Ibin_uT32__mrp2_simulator__ubuntu_trusty_i386__binary/builds/1/log:Error removing intermediate container 9457f5cd49f7: rmdriverfs: Driver aufs failed to remove root filesystem 9457f5cd49f71d10a7ce6322f148d76e753a612205534c41c65d027264cacb69: rename /var/lib/docker/aufs/diff/9457f5cd49f71d10a7ce6322f148d76e753a612205534c41c65d027264cacb69 /var/lib/docker/aufs/diff/9457f5cd49f71d10a7ce6322f148d76e753a612205534c41c65d027264cacb69-removing: device or resource busy
Isrc_uS__mrp2_viz__ubuntu_saucy__source/builds/1/log:Error removing intermediate container d9eb47796e49: rmdriverfs: Driver aufs failed to remove root filesystem d9eb47796e49f34e660ba5b31e629a701e419435fbe8a970c930421187818210: rename /var/lib/docker/aufs/mnt/d9eb47796e49f34e660ba5b31e629a701e419435fbe8a970c930421187818210 /var/lib/docker/aufs/mnt/d9eb47796e49f34e660ba5b31e629a701e419435fbe8a970c930421187818210-removing: device or resource busy

From the snippet it can be seen that the error occurs with a variety of dockerfiles. The line on which it fails is also quite diverse. Many are quick running commands like CMD ENV or echo but some are apt-get invocations which take much longer.

This was discovered in ros-infrastructure/ros_buildfarm#134 and also might suggest reverting ros-infrastructure/buildfarm_deployment#80

@guenhter

This comment has been minimized.

guenhter commented Feb 8, 2016

Also experience this problem on Debian 7. I also can reproduce it after reboot.

Error:
docker rm agent2 Failed to remove container (agent2): Error response from daemon: Driver aufs failed to remove root filesystem f6a8abcb5d0b398fc8808f2ef426414b7e8bafe0852e14732cf2e3e3b32d66da: layer not retained

Docker version
`Client:
Version: 1.10.0
API version: 1.22
Go version: go1.5.3
Git commit: 590d510
Built: Thu Feb 4 18:25:18 2016
OS/Arch: linux/amd64

Server:
Version: 1.10.0
API version: 1.22
Go version: go1.5.3
Git commit: 590d510
Built: Thu Feb 4 18:25:18 2016
OS/Arch: linux/amd64`

Docker Info
Containers: 7 Running: 6 Paused: 0 Stopped: 1 Images: 193 Server Version: 1.10.0 Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 208 Dirperm1 Supported: true Execution Driver: native-0.2 Logging Driver: json-file Plugins: Volume: local Network: bridge null host Kernel Version: 3.16.0-0.bpo.4-amd64 Operating System: Debian GNU/Linux 7 (wheezy) OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 31.45 GiB Name: chell ID: BXX3:THMK:SWD4:FP35:JPVM:3MV4:XJ7S:DREY:O6XO:XYUV:RHXO:KUBS WARNING: No memory limit support WARNING: No swap limit support WARNING: No oom kill disable support WARNING: No cpu cfs quota support WARNING: No cpu cfs period support

uname -a
Linux chell 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3~bpo70+1 (2016-01-19) x86_64 GNU/Linux

@claytono

This comment has been minimized.

claytono commented Feb 8, 2016

Still seeing this issue on 1.10, but the error message is different now:

Failed to remove container (designate-api): Error response from daemon: Unable to remove filesystem for 4ad829a0e8a5bb45add135a7e1cfee323ffa378f9b9a3f09d36d939230137f1f: remove /var/lib/docker/containers/4ad829a0e8a5bb45add135a7e1cfee323ffa378f9b9a3f09d36d939230137f1f/mqueue: device or resource busy

Also, on 1.8.1 the container name would be changed. now I have to rm -f the container to be able to reuse the container name.

docker info:

Containers: 10
 Running: 9
 Paused: 0
 Stopped: 1
Images: 3
Server Version: 1.10.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 54
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: son-file
Plugins:
 Volume: local
 Network: null host
Kernel Version: 3.13.0-67-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.548 GiB
Name: dev02-control-001
ID: W27R:L56T:ZR74:VKCQ:RYT5:JSIT:PRJL:CT4A:N4UI:CBBU:2LGP:KUAZ
WARNING: No swap limit support

claytono pushed a commit to twc-openstack/garethr-docker that referenced this issue Feb 15, 2016

Clayton O'Neill
Work around docker rm errors
This is a hopefully temporary work around this issue:
moby/moby#9665

This means that docker rm is unreliable.  All we really care about when
we stop the service is to ensure that the service is stopped, so we
explicitly ask docker for that via inspect instead of relying on the rm
result code.
@commandgjj

This comment has been minimized.

commandgjj commented Mar 1, 2016

@mimiyan I reboot the computer according to your suggestion. It indeed resolved the "dead" container can not removed issue. Thanks!

@jeekajoo

This comment has been minimized.

jeekajoo commented Mar 8, 2016

Same as @Dvorak with docker 1.10.2 and aufs storage.
Force removing the container did the trick for me.

@devminz

This comment has been minimized.

devminz commented Mar 20, 2016

Faced similar issue in deleting an old docker images directory ,
not in use by new docker-machine installation.

$ sudo rm -rf  docker_images.old/
rm: cannot remove ‘docker_images.old/aufs’: Device or resource busy 

Solved by :

$ sudo service docker stop 
$ sudo modprobe -r aufs  
$ sudo rm -rf  docker_images.old/

@GameScripting

This comment has been minimized.

GameScripting commented May 25, 2016

Found a workarround for the problem, at least when used with docker-compose see #3786 (comment)

claytono pushed a commit to twc-openstack/garethr-docker that referenced this issue Jul 12, 2016

Clayton O'Neill
Work around docker rm errors
This is a hopefully temporary work around this issue:
moby/moby#9665

This means that docker rm is unreliable.  All we really care about when
we stop the service is to ensure that the service is stopped, so we
explicitly ask docker for that via inspect instead of relying on the rm
result code.
@chadxz

This comment has been minimized.

chadxz commented Aug 3, 2016

i had run into this issue as well. removing the /:/rootfs:ro volume works around the issue for me.

Docker 1.11.2, cAdvisor 0.23.1

@ko-christ

This comment has been minimized.

ko-christ commented Dec 16, 2016

Run into this device or resource busy error on ansible managed containers (on a re-create task), leaving containers Dead.

ansible/API log

Error removing container cc59a6ac0f600b4e98d6968a5c1c8d2b1277e95d0340a7400277920c0f28a570: 500 Server Error: Internal Server Error (\"{\"message\":\"Driver aufs failed to remove root filesystem cc59a6ac0f600b4e98d6968a5c1c8d2b1277e95d0340a7400277920c0f28a570: rename /docker/951968.951968/aufs/mnt/2a47f9db4760ad1880d862ab9d478703ce47e1a34109a042a0efc44dbb2d0357 /docker/951968.951968/aufs/mnt/2a47f9db4760ad1880d862ab9d478703ce47e1a34109a042a0efc44dbb2d0357-removing: device or resource busy"

journalctl docker log (same error)

level=error msg="Handler for DELETE /v1.24/containers/cc59a6ac0f600b4e98d6968a5c1c8d2b1277e95d0340a7400277920c0f28a570 returned error: Driver aufs failed to remove root filesystem cc59a6ac0f600b4e98d6968a5c1c8d2b1277e95d0340a7400277920c0f28a570: rename /docker/951968.951968/aufs/mnt/2a47f9db4760ad1880d862ab9d478703ce47e1a34109a042a0efc44dbb2d0357 /docker/951968.951968/aufs/mnt/2a47f9db4760ad1880d862ab9d478703ce47e1a34109a042a0efc44dbb2d0357-removing: device or resource busy"

using latest stable 1.12.4.

docker version
Client:
 Version:      1.12.4
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   1564f02
 Built:        Mon Dec 12 23:40:33 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.4
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   1564f02
 Built:        Mon Dec 12 23:40:33 2016
 OS/Arch:      linux/amd64
@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Dec 16, 2016

@ko-christ there are many reasons for that to happen, but basically it comes down to mounts leaking (running cAdvisor, which bind-mounts various directories, is a known cause for this). There is still an open issue tracking this, see #21704, which may contain useful information.

I'm locking this issue, because this issue is closed, and predates several issues that are still open.

@moby moby locked and limited conversation to collaborators Dec 16, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.