Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[aufs] Unable to delete a container when / partition is mounted #18509

Closed
jeanpralo opened this issue Dec 8, 2015 · 37 comments
Closed

[aufs] Unable to delete a container when / partition is mounted #18509

jeanpralo opened this issue Dec 8, 2015 · 37 comments
Labels
area/storage/aufs kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. version/1.9

Comments

@jeanpralo
Copy link

Hi guys,

BUG REPORT INFORMATION

docker version:
Client:
Version: 1.9.0
API version: 1.21
Go version: go1.4.2
Git commit: 76d6bc9
Built: Tue Nov 3 17:43:42 UTC 2015
OS/Arch: linux/amd64

Server:
Version: 1.9.0
API version: 1.21
Go version: go1.4.2
Git commit: 76d6bc9
Built: Tue Nov 3 17:43:42 UTC 2015
OS/Arch: linux/amd64

docker info:
Containers: 11
Images: 181
Server Version: 1.9.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 205
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-43-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 1
Total Memory: 987.9 MiB
Name: ch-lundi3
ID: CVYX:O4VO:FX57:K7JU:PG6N:DZRV:GXW7:XN2E:4LUO:QSF4:B2YJ:CYRZ
WARNING: No swap limit support

uname -a :
Linux ch-lundi3 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce the problem

1- docker run -d --name lafauteaqui -v /:/check:ro ubuntu:14.04 /bin/sh -c 'while true; do echo la_faute_a_flop; sleep 1; done'
2- docker run -it --rm ubuntu:14.04
3- docker restart lafauteaqui
4- Try to exit the shell in the container started during step2:

Error deleting container: Error response from daemon: Driver aufs failed to remove root filesystem 3d1e082f48eb875d3f2117342b082e1aca84164bfa7ccf253c53e0dc38f61e77: rename /var/lib/docker/aufs/mnt/3d1e082f48eb875d3f2117342b082e1aca84164bfa7ccf253c53e0dc38f61e77 /var/lib/docker/aufs/mnt/3d1e082f48eb875d3f2117342b082e1aca84164bfa7ccf253c53e0dc38f61e77-removing: device or resource busy

Describe the results you received:
It should exit gracefully

@jeanpralo
Copy link
Author

I have also tested it in docker 1.9.1 and it is a problem as well.

@thaJeztah
Copy link
Member

Thanks for reporting @jeanpralo, just to check; is this a regression since 1.8? Were you able to do this in 1.8 without issues?

@jeanpralo
Copy link
Author

@thaJeztah I have just been able to reproduce using docker 1.8.3 and 1.7.1 so it has been around for quite a while now.

@thaJeztah
Copy link
Member

@jeanpralo thanks for checking.

@thaJeztah
Copy link
Member

In that output; is 3d1e082f48eb875d3f2117342b082e1aca84164bfa7ccf253c53e0dc38f61e77 the ID of the first container (lafauteaqui) or the second one (started in step 2)?

@jeanpralo
Copy link
Author

@thaJeztah it's the id of the second container started in step 2

@jeanpralo
Copy link
Author

@thaJeztah will you be assigning any labels on that issue ?

Just want to know what sort of timeframe it will be consider to fix that issue as it is quite a problem especially when using a container to get metrics from servers ( i use collectd in a container to get metrics out of my servers so for the df plugin i need to mount / ) :)

Thank you :)

@thaJeztah thaJeztah added the kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. label Dec 14, 2015
@thaJeztah
Copy link
Member

@jeanpralo I added a "bug" label. I can't give you an estimate as to when this will be looked into.

I just tried to reproduce, but I'm unable to reproduce this 😢 I tried these steps;

$ docker run -d --name looper -v /:/check:ro ubuntu /bin/sh -c 'while true; do echo foo; sleep 1; done'
60233ec48b4d64456405ec6a633270594a9964f70e969e0b2432075158e5c162
$ docker run -it --rm ubuntu
root@df9fa8750296:/#

in a second shell:

$ docker restart looper
looper

first shell:

root@df9fa8750296:/# exit
exit

@jeanpralo
Copy link
Author

@thaJeztah are you using aufs or devicemapper as a storage driver.

I have been able to reproduce that problem only with aufs so far, no problem with devicemapper.

@jeanpralo jeanpralo changed the title Unable to delete a container when / partition is mounted [aufs] Unable to delete a container when / partition is mounted Dec 14, 2015
@thaJeztah
Copy link
Member

That was on aufs. I can try it on some other hosts

@jeanpralo
Copy link
Author

@thaJeztah I was able to reproduce on aws ec2 choosing the latest ubuntu AMI.

From a fresh install i ran those commands:

$ history 
    1  curl
    2  curl -sSL https://get.docker.com/ | sh
    3  docker info
    4  screen

Inside the screen I run:

1- Create the looping container mounting /

$ docker run -d --name looper -v /:/check:ro ubuntu /bin/sh -c 'while true; do echo foo; sleep 1; done'
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
0bf056161913: Pull complete
1796d1c62d0c: Pull complete
e24428725dd6: Pull complete
89d5d8e8bafb: Pull complete
Digest: sha256:a2b67b6107aa640044c25a03b9e06e2a2d48c95be6ac17fb1a387e75eebafd7c
Status: Downloaded newer image for ubuntu:latest
4eccf7ded478d22b2c4dda1bddcb890a6572dc7844493c40a7e635526e187e64

2- Start a fresh container with a shell interactively

$ docker run -it --rm ubuntu
root@a484eeeda7e5:/#

3- Restart the looper container

$ docker restart looper

4- Try to exit and it fails

root@a484eeeda7e5:/# exit
Error deleting container: Error response from daemon: Driver aufs failed to remove root filesystem a484eeeda7e52a27aef973fcb1b0a15cba28470b68e459341e071282dd6dc048: rename /var/lib/docker/aufs/mnt/a484eeeda7e52a27aef973fcb1b0a15cba28470b68e459341e071282dd6dc048 /var/lib/docker/aufs/mnt/a484eeeda7e52a27aef973fcb1b0a15cba28470b68e459341e071282dd6dc048-removing: device or resource busy

Note that the container stays with a status Dead and to be able to delete it I need to stop/start the looper container or restart it:

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
a484eeeda7e5        ubuntu              "/bin/bash"              6 minutes ago       Dead                                    
4eccf7ded478        ubuntu              "/bin/sh -c 'while tr"   6 minutes ago       Up 6 minutes                            looper
$ docker rm a484eeeda7e5
Error response from daemon: Driver aufs failed to remove root filesystem a484eeeda7e52a27aef973fcb1b0a15cba28470b68e459341e071282dd6dc048: rename /var/lib/docker/aufs/mnt/a484eeeda7e52a27aef973fcb1b0a15cba28470b68e459341e071282dd6dc048 /var/lib/docker/aufs/mnt/a484eeeda7e52a27aef973fcb1b0a15cba28470b68e459341e071282dd6dc048-removing: device or resource busy
Error: failed to remove containers: [a484eeeda7e5]
$ docker restart looper
$ docker rm a484eeeda7e5
a484eeeda7e5
$

docker info:
docker info
Containers: 2
Images: 4
Server Version: 1.9.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 8
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-48-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 1
Total Memory: 992.5 MiB
Name: ip-172-31-4-12
ID: BXE7:KCRR:QQ44:X66G:RYIB:JGE6:VS2N:E76E:OASO:VAAM:GV4P:VJN5
WARNING: No swap limit support

@cpuguy83
Copy link
Member

I imagine this is because there are mounts leaking into the container by mounting in /.

@jeanpralo
Copy link
Author

So basically we need to avoid to mount anything which is /var/lib/docker related ?

Is this something fixable or we need to find a workaround to that problem, sorry I have not read much about aufs.

@cpuguy83
Copy link
Member

I don't think this is AUFS related in particular.
I think this was fixed by #18076

Can you try with master?

@jeanpralo
Copy link
Author

@cpuguy83 I am testing with the latest compiled binary from https://master.dockerproject.org/, never really debugged in docker before so tell me if I have to compile it directly from master.

The problem is not exactly the same anymore seems like there is a problem with the shared memory mountpoint now:

Error deleting container: Error response from daemon: Unable to remove filesystem for c9bca1fd0fd42817fd55e0d6f50019256e14854adbafb91cfae393f88bd3b90d: remove /var/lib/docker/containers/c9bca1fd0fd42817fd55e0d6f50019256e14854adbafb91cfae393f88bd3b90d/shm: device or resource busy

I have followed the exact same step as before.

./docker-1.10.0-dev info:
Containers: 2
Images: 1
Server Version: 1.10.0-dev
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 8
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.13.0-43-generic
Operating System: Ubuntu 14.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.455 GiB
Name: dockertest3
ID: Y5UR:44DG:M4D5:726Q:W42V:WIHN:IS35:AVDV:QIX4:FXRJ:T2XS:EPD4
WARNING: No swap limit support

@kforner
Copy link

kforner commented Dec 23, 2015

I'm getting the same bug.
This is really annoying since it prevents the server from shutting down.
Is there some work-around ? How to manually umount those aufs mnts ?

docker info:
Containers: 1
Images: 99
Server Version: 1.9.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 103
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-42-generic
Operating System: Ubuntu 14.04.3 LTS

@sjackman
Copy link

I've just encountered this bug as well. Is there a workaround?

❯❯❯ docker pull sjackman/linuxbrew-standalone
Using default tag: latest
latest: Pulling from sjackman/linuxbrew-standalone
8cce2451c588: Pulling fs layer 
Pulling repository docker.io/sjackman/linuxbrew-standalone
12d3f2521583: Error pulling image (latest) from docker.io/sjackman/linuxbrew-sta12d3f2521583: Error pulling image (latest) from docker.io/sjackman/linuxbrew-standalone, mkdir /mnt/sda1/var/lib/docker/graph/_tmp/6375f193d455c206cfe08a0c71664Error pulling image (latest) from docker.io/sjackman/linuxbrew-standalone, mkdir /mnt/sda1/var/lib/docker/graph/_tmp/6375f193d455c206cfe08a0c7166447a4b9c83469378b57424ec15576217d235: read-only file system'
❯❯❯ docker --version
Docker version 1.9.1, build a34a1d5

@thaJeztah
Copy link
Member

@sjackman that looks like a whole different issue, more likely something on your system;

mkdir /mnt/sda1/var/lib/docker/graph/_tmp/6375f193d455c206cfe08a0c7166447a4b9c83469378b57424ec15576217d235:
read-only file system

@sjackman
Copy link

Rebooting the virtual machine resolved the issue.

@evgnomon
Copy link

+1
worked for me:
sudo systemctl restart docker

$docker version
Client:
Version: 1.10.1
API version: 1.22
Go version: go1.5.3
Git commit: 9e83765-unsupported
Built: Sat Feb 20 14:47:15 2016
OS/Arch: linux/amd64

Server:
Version: 1.10.1
API version: 1.22
Go version: go1.5.3
Git commit: 9e83765-unsupported
Built: Sat Feb 20 14:47:15 2016
OS/Arch: linux/amd64

$uname -a
Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u5 (2015-10-09) x86_64 GNU/Linux

@thaJeztah
Copy link
Member

@ghasemz looks like you're not running an official binary; it's built from modified source (see "git commit" in your output); did you build it yourself?

@jeanpralo
Copy link
Author

@thaJeztah have you been able to reproduce on aws ? Just wondering if I am the only one having the issue so far :)

@kabobbob
Copy link

kabobbob commented Mar 7, 2016

+1 for receiving this error as well.

$docker version
Client:
Version: 1.10.2
API version: 1.22
Go version: go1.5.3
Git commit: c3959b1
Built: Mon Feb 22 21:37:01 2016
OS/Arch: linux/amd64

Server:
Version: 1.10.2
API version: 1.22
Go version: go1.5.3
Git commit: c3959b1
Built: Mon Feb 22 21:37:01 2016
OS/Arch: linux/amd64

$docker info
Containers: 10
Running: 8
Paused: 0
Stopped: 2
Images: 11
Server Version: 1.10.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 140
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.16.0-55-generic
Operating System: Ubuntu 14.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.67 GiB
Name: [SERVER_NAME]
ID: TOX5:Y4KU:K7G5:XO3Q:OMGT:PW3V:OJRP:WJJT:PU6C:OXBV:QF75:PKJB
WARNING: No swap limit support

$docker rm 67bfeb577c18
Failed to remove container (67bfeb577c18): Error response from daemon: Driver aufs failed to remove root filesystem 67bfeb577c187c2ff3775dfd3acbd5090c8e53a92b614345f6545633d0bf3d49: rename /var/lib/docker/aufs/mnt/29f0bc9f166e3951d178b9b8df661da3a8a74bc9595bf0bdd740d1cac86d31b1 /var/lib/docker/aufs/mnt/29f0bc9f166e3951d178b9b8df661da3a8a74bc9595bf0bdd740d1cac86d31b1-removing: device or resource busy

@jeanpralo I was getting the shared memory bug you mentioned above and just like your output, I was on kernel 3.13. I upgraded to 3.16, uninstalled/reinstalled docker, and now started receiving this error.

@kabobbob
Copy link

kabobbob commented Mar 8, 2016

Also to note, I'm running all my containers on a named network. I wonder if this issue is due to linking between the containers? Maybe this was already mentioned.

@jordic
Copy link

jordic commented Mar 27, 2016

I also have the same problem:

Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 15:54:52 2016
OS/Arch: linux/amd64

Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 15:54:52 2016
OS/Arch: linux/amd64

Ubuntu 14

Can't remove some containers without restarting the main docker profess. And when its restarted, seems like containers are garbage collected. Any workaround, or solution?

@kabobbob
Copy link

@jordic out of curiosity, what kernel version are you running?

@jordic
Copy link

jordic commented Mar 29, 2016

3.13.0-79-generic

@kabobbob
Copy link

@jordic at one time I thought it had to do with the kernel version, but now, I'm not so sure. We're on 3.16 and sometimes still experience this issue.

@Teudimundo
Copy link

Same issue here:

Docker Client 1.11.0
Docker Server 1.11.0
Ubuntu 14.04.4 LTS (Kernel: 3.13.0-85-generic)

The container is marked as Dead. I don't know if it can help, but when I try to restart it with docker-compose up I get:

ERROR: open 
/var/lib/docker/containers/c1398f3be5333707a6710c9f4f811f56b515ef2934d6016348117bd7c894f547/config.v2.json: no such file or directory

@jordic
Copy link

jordic commented Apr 20, 2016

@Teudimundo are you running cadvisor on the docker host?

@Teudimundo
Copy link

No cadvisor, that container in particular was just a nginx based on the nginx:1.8.1-alpine using a local named volume started through swarm using docker-compose.

@Teudimundo
Copy link

Teudimundo commented Apr 21, 2016

@jordic I think I found the main reason for my problem, it is related to stopping a container in Restarting phase, here's the bug I filed:
#22222

@gtron
Copy link

gtron commented May 25, 2016

Same issue with version Docker version 1.10.3, build 20f81dd

Solved unmounting the dir:
umount /var/lib/docker/aufs/mnt/197c9141c021c89569db52218795528ae6621ebeea3cc5403613c97b56fe8720

I'm not sure it's ok what I did ... but it worked! ;)

@firelyu
Copy link

firelyu commented Jun 2, 2016

Same issue in docker 1.10.3 on Ubuntu 14.04 64 bits. The kernel is 3.13.0-65-generic.

# docker-compose down
Stopping gcplatform_gcplatform_1 ... done
Removing gcplatform_gcplatform_1 ... error

ERROR: for gcplatform_gcplatform_1  Driver aufs failed to remove root filesystem 005d7056a826d9545fafeae490f44c147c7eaf0e77643c3d08bc6f81a6a1f24e: rename /var/lib/docker/aufs/mnt/019022701b39ed9f583da52a430a96c3275097e82c0cd323ce490aaf99f52970 /var/lib/docker/aufs/mnt/019022701b39ed9f583da52a430a96c3275097e82c0cd323ce490aaf99f52970-removing: device or resource busy
# docker version
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64
# docker info
Containers: 21
 Running: 7
 Paused: 0
 Stopped: 14
Images: 386
Server Version: 1.10.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 362
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.13.0-65-generic
Operating System: Ubuntu 14.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.955 GiB
Name: gcdev-hz-01
ID: EAUO:RXS5:MIMG:RHLH:7S4S:YWVI:YLPZ:DV2Q:RU4N:U745:VN52:2LRD
WARNING: No swap limit support

@marcellodesales
Copy link

marcellodesales commented Jun 2, 2016

I'm having the same problem on RHEL 7.2.

Error

[root@pe2enpmas300 npmo-server]# docker-compose -f services/splunk.yml rm
WARNING: Not including one-off containers created by `docker-compose run`.
To include them, use `docker-compose rm --all`.
This will be the default behavior in the next version of Compose.

Going to remove services_splunkforwarder_1
Are you sure? [yN] y
Removing services_splunkforwarder_1 ... error

ERROR: for services_splunkforwarder_1  Unable to remove filesystem for 
39a48b71ffaa043c82612016ea54ff6e2aec6432a6d3081fd9b15f6f5cda8775: remove
 /var/lib/docker/containers/39a48b71ffaa043c82612016ea54ff6e2aec6432a6d3081fd9b15f6f5cda8775/shm: device or resource busy

Can't unmount that...

[root@pe2enpmas300 npmo-server]# umount /var/lib/docker/containers/39a48b71ffaa043c82612016ea54ff6e
2aec6432a6d3081fd9b15f6f5cda8775/shm
umount: /var/lib/docker/containers/39a48b71ffaa043c82612016ea54ff6e
2aec6432a6d3081fd9b15f6f5cda8775/shm: not mounted

OS

[root@pe2enpmas300 npmo-server]# uname -a
Linux pe2enpmas300.corp.comp.net 3.10.0-327.3.1.el7.x86_64 #1 SMP Fri Nov 20 05:40:26 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

Docker Info

[root@pe2enpmas300 npmo-server]# docker version
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:39:25 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:39:25 2016
 OS/Arch:      linux/amd64

@ryanlovett
Copy link

This happened to me on Ubuntu 14.04 with docker-compose when I attempted to bind mount the volume /dev/log:/dev/log:rw.

$ docker-compose --version
docker-compose version 1.8.0, build f3628c7
$ docker --version
Docker version 1.12.0, build 8eab29e
$ docker-compose up
Removing nginx_n0_1
ERROR: Driver aufs failed to remove root filesystem 37e33e150f12e23b95b0b500e7d5743e3f10fe4008547414118f74faa83ea96c: rename /var/lib/docker/aufs/diff/366edeb8a8d864a60160ce1011ce5b91df79f02bfd876fde3b50b80e9aed46eb /var/lib/docker/aufs/diff/366edeb8a8d864a60160ce1011ce5b91df79f02bfd876fde3b50b80e9aed46eb-removing: device or resource busy

@thaJeztah
Copy link
Member

The AuFS storage driver has been deprecated and has been removed from our code base (#45342) for the upcoming v24.0.0 release, so we're closing tickets related to AuFS

@thaJeztah thaJeztah closed this as not planned Won't fix, can't repro, duplicate, stale Apr 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/storage/aufs kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. version/1.9
Projects
None yet
Development

No branches or pull requests