Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Sometimes container's root filesystem is kept mounted preventing docker rm on btrfs #4303

Closed
farcaller opened this Issue · 15 comments

9 participants

@farcaller

Note: this might be a side effect of same problem that results in #4295.

Problem description: sometimes container rootfs is kept mounted after container exited, preventing its removal.

# docker ps -a|grep 754
754d7d92f1dd        cellofellow/rpi-arch:latest   /bin/bash           4 hours ago         Exit 0                                  loving_bell

# docker rm 754d7d92f1dd
Error: container_delete: Cannot destroy container 754d7d92f1dd: Driver btrfs failed to remove root filesystem 754d7d92f1ddfca4ec8b822910465d092804c5847636963ce844be15147e6b1e: Failed to destroy btrfs snapshot: device or resource busy
2014/02/24 12:52:11 Error: failed to remove one or more containers

# mount|grep 754d7d92f1dd
/dev/mmcblk1p2 on /var/lib/docker/containers/754d7d92f1ddfca4ec8b822910465d092804c5847636963ce844be15147e6b1e/root type btrfs (rw,relatime,compress=lzo,ssd,discard,space_cache)
/dev/mmcblk1p2 on /var/lib/docker/containers/754d7d92f1ddfca4ec8b822910465d092804c5847636963ce844be15147e6b1e/root/.dockerinit type btrfs (ro,relatime,compress=lzo,ssd,discard,space_cache)
/dev/mmcblk1p2 on /var/lib/docker/containers/754d7d92f1ddfca4ec8b822910465d092804c5847636963ce844be15147e6b1e/root/.dockerenv type btrfs (ro,relatime,compress=lzo,ssd,discard,space_cache)
/dev/mmcblk1p2 on /var/lib/docker/containers/754d7d92f1ddfca4ec8b822910465d092804c5847636963ce844be15147e6b1e/root/etc/resolv.conf type btrfs (ro,relatime,compress=lzo,ssd,discard,space_cache)
/dev/mmcblk1p2 on /var/lib/docker/containers/754d7d92f1ddfca4ec8b822910465d092804c5847636963ce844be15147e6b1e/root/etc/hostname type btrfs (ro,relatime,compress=lzo,ssd,discard,space_cache)
/dev/mmcblk1p2 on /var/lib/docker/containers/754d7d92f1ddfca4ec8b822910465d092804c5847636963ce844be15147e6b1e/root/etc/hosts type btrfs (ro,relatime,compress=lzo,ssd,discard,space_cache)

# umount -R /var/lib/docker/containers/754d7d92f1ddfca4ec8b822910465d092804c5847636963ce844be15147e6b1e/root

# docker rm 754d7d92f1dd
754d7d92f1dd

I suppose, adding rootfs unmount verification to rm operation would do the trick.

@unclejack
Collaborator

@farcaller We need more details in order to be able to investigate. Please provide the following: the output of uname -a, docker version, docker info, the name and version of the Linux distribution you were using when running into these problems.

@farcaller
$ uname -a
Linux alarm 3.13.4-1-VF #2 SMP Sat Feb 22 22:56:54 UTC 2014 armv7l GNU/Linux

$ docker version
Client version: 0.8.1
Go version (client): go1.2
Git commit (client): a1598d1-dirty
Server version: 0.8.1
Git commit (server): a1598d1-dirty
Go version (server): go1.2
Last stable version: 0.8.1

$ docker info
Containers: 3
Images: 12
Driver: btrfs
Debug mode (server): true
Debug mode (client): false
Fds: 21
Goroutines: 25
Execution Driver: lxc-0.9.0
EventsListeners: 10
Kernel Version: 3.13.4-1-VF
Init SHA1: 5d12d994969051f9ebca5a65c0d5f7d8bd3ce0a3
Init Path: /usr/lib/docker/dockerinit
WARNING: No swap limit support

Running Arch on ARM (apparently I patched docker to be running on ARM). I know this is not really the supported distro/arch combo, but this problem doesn't look exactly arch-relevant to me.

@vieux
Owner

I'm closing this for now, ARM is not supported ATM. If somebody encounter this issue on a supported arch, please comment here.

@vieux vieux closed this
@stephenmw

Please reopen. I ran into the same problem using docker in x86_64 vagrant (precise). I used a modified version of the vagrant file that comes with docker so I have the latest docker and the 3.8 kernel on ubuntu precise.

@wking
@farcaller

Not related to #4100, not a dup either (I own both bugs :wink: ). This one is about container root and mount binds being alive after container is stopped and btrfs driver failing to un mount them before subvol removal. #4100 is about creating btrfs subvols from inside the container, that breaks container removal as well.

@wking
@tianon tianon reopened this
@crosbymichael

@farcaller can you try to reproduce on master. We moved the bind mounts into the mount namespace so the kernel cleans up things for us now. It has fixed various issues around docker rm due to mounts.

@farcaller

Will do.

@csirac2

Related to #4897 ? I just left a comment there with what seems to be the same problem.

@wking
@vieux
Owner

@farcaller any news on 0.11.1 ?

@vieux vieux removed the /project/point-o label
@csirac2

Thanks to this bug I've fixed my own problems with a sleep 3 strategically placed between docker cp and docker rm statements in my build scripts - my adventure is documented in #4897

@unclejack
Collaborator

This has been fixed.

@unclejack unclejack closed this
@jkp
jkp commented

I don't think this is fixed - it is happening consistently for me on circle-ci. They have a custom build of 1.2.0 it seems:

ubuntu@box928:~$ docker -v
Docker version 1.2.0-circleci-lxc-b, build 4c562a1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.