Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker does not remove btrfs subvolumes when destroying container #9939

Closed
phemmer opened this issue Jan 7, 2015 · 40 comments

Comments

Projects
None yet
@phemmer
Copy link
Contributor

commented Jan 7, 2015

I receive the following error when deleting a container which created a btrfs subvolume (as happens when you run docker in docker).

# docker run --rm fedora:20 sh -c 'yum -y -q install btrfs-progs && btrfs subvolume create /test'
Public key for lzo-2.08-1.fc20.x86_64.rpm is not installed
Public key for e2fsprogs-libs-1.42.8-3.fc20.x86_64.rpm is not installed
Importing GPG key 0x246110C1:
 Userid     : "Fedora (20) <fedora@fedoraproject.org>"
 Fingerprint: c7c9 a9c8 9153 f201 83ce 7cba 2eb1 61fa 2461 10c1
 Package    : fedora-release-20-3.noarch (@fedora-updates/$releasever)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-x86_64
Create subvolume '//test'
FATA[0033] Error response from daemon: Cannot destroy container c9badf5fc87bb9bfb50a3ee6e5e7c840476bd704e62404c9136aab4d27239d1e: Driver btrfs failed to remove root filesystem c9badf5fc87bb9bfb50a3ee6e5e7c840476bd704e62404c9136aab4d27239d1e: Failed to destroy btrfs snapshot: directory not empty 

Info:

# docker info
docContainers: 22
Images: 47
Storage Driver: btrfs
Execution Driver: native-0.2
Kernel Version: 3.13.2-gentoo
Operating System: Gentoo/Linux
CPUs: 8
Total Memory: 15.64 GiB
Name: whistler
ID: RL3I:O6RS:UJRN:UU74:WAGE:4X5B:T2ZU:ZRSU:BN6Q:WN7L:QTPM:VCLN
Username: phemmer
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support

# docker version
Client API version: 1.16
Go version (client): go1.3.3
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8
@tianon

This comment has been minimized.

Copy link
Member

commented Jan 7, 2015

@jfrazelle didn't you look into this? Did you get anywhere good? :)

@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2015

still looking i really want to fix this, but it is a permissions thing, because we are making the same exact syscall as btrfs-tools

@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2015

but this is a duplicate issue

@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2015

duplicate of #7773, closing, lmk tho if you believe differently

@jessfraz jessfraz closed this Jan 7, 2015

@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2015

actually I like your better, I'm going to close the other

@tianon

This comment has been minimized.

Copy link
Member

commented Jan 9, 2015

This is something else, @jfrazelle. 😄

This is sub-subvolumes with BTRFS (ie, docker-in-docker artifacts, etc). This is why we had to add sort -r to our nuke script (so that it'd delete the inner subvolumes before the outer ones).

@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Jan 13, 2015

@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Jan 13, 2015

working on a port

@jessfraz jessfraz self-assigned this Jan 13, 2015

@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Jan 13, 2015

but if someone else who is super good at C wants to do it go ahead :P

@rgbkrk

This comment has been minimized.

Copy link

commented Mar 3, 2015

@tianon What are you using for a nuke script now?

@tianon

This comment has been minimized.

@rgbkrk

This comment has been minimized.

Copy link

commented Mar 6, 2015

Awesome, thank you!

@prateeksahu

This comment has been minimized.

Copy link

commented May 5, 2016

Does any one know of any workaround for it? the btrfs seems to be corrupted for me and somehow i need to make atleast the machine filesystem work.

@InAnimaTe

This comment has been minimized.

Copy link

commented Jun 1, 2016

To anyone else having to derp with this, I had to do similar to @TomasTomecek

┌[root@lovell] 
└[/var/lib/docker]> btrfs subvolume delete btrfs/subvolumes/*

^that worked for me ;)

@thechane

This comment has been minimized.

Copy link

commented Jul 6, 2016

Anyone know a way to distinguish the orphan subvolumes? I don't want to blow away my running containers.

@RomainVernoux

This comment has been minimized.

Copy link

commented Feb 15, 2017

I encountered the issue in v1.13 in my CI environment. I eventually figured out that it occured when multiple Jenkins jobs were doing docker system prune simultaneously on a btrfs filesystem. Is there any doc or information regarding thread-safety of the docker daemon?

phoolish added a commit to tablexi/chef-sendmail-ses that referenced this issue Feb 17, 2017

phoolish added a commit to tablexi/chef-sendmail-ses that referenced this issue Feb 17, 2017

@hipposareevil

This comment has been minimized.

Copy link

commented Mar 6, 2017

@thechane
I wrote a script that traverses the docker structures and finds what directories are orphaned and optionally deletes them. If it'd be useful to people I can put up on github and share here. I've only done basic testing on oraclelinux but it should apply to other OSes.

@justincormack

This comment has been minimized.

Copy link
Contributor

commented Mar 6, 2017

@julienlau

This comment has been minimized.

Copy link

commented Aug 22, 2017

Fix embedded in "docker clean" would be nice

@huegelc

This comment has been minimized.

Copy link

commented Sep 21, 2017

still hitting this bug. btrfs/subvolumes grows very fast.

@johnharris85

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2017

@hipposareevil if you can post that script it would be super useful please :)

@hipposareevil

This comment has been minimized.

Copy link

commented Sep 25, 2017

@johnharris85 Turns out that my script would negatively affect the system as it was deleting subvolumes that were tied to images. So if you tried to remove an image, it would be broken. It would work while you were running things, but then later when manipulating images you'd be hosed. :|

@johnharris85

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2017

OK thanks @hipposareevil, anyway to filter out those volumes that were being used by images as well? Mind posting the script anyway (even as a gist) so I can hack on it?

@hipposareevil

This comment has been minimized.

Copy link

commented Oct 3, 2017

@johnharris85 Shoot me an email at dockergist@mailinator.com

@einyx

This comment has been minimized.

Copy link

commented Nov 6, 2017

This is still an issue 😢

@piahoo

This comment has been minimized.

Copy link

commented Nov 28, 2017

my subvolumes on btrfs grows to 222gb

@huegelc

This comment has been minimized.

Copy link

commented Nov 28, 2017

I´ve deactivated btrfs for now and I´m using overlay2

@julienlau

This comment has been minimized.

Copy link

commented Dec 4, 2017

with docker 1.13+ you have the command below that can help:
docker system prune -f

@JonasT

This comment has been minimized.

Copy link

commented Feb 3, 2018

Since people here say this still happens, maybe it might be a good idea to reopen this ticket?

@beerendlauwers

This comment has been minimized.

Copy link

commented Mar 21, 2018

@huegelc Does the overlay2 driver work on a btrfs file system? The documentation says only ext4 and xfs are supported.

@devopxy

This comment has been minimized.

Copy link

commented Aug 14, 2018

Using btrfs commands I could remove those sub-volumes e.g. :

btrfs subvolume delete eb669bae4f4aa17f3c432d956f481146e4ac77e3f1803fee15e1f2b17787510d-init

@JonasT

This comment has been minimized.

Copy link

commented Aug 21, 2018

@devopxy yea that works, but it doesn't solve the underlying problem... what I usually do is uninstall docker, then wipe /var/lib/docker completely with all files and btrfs subvolumes inside, then reinstall.

@tmalavash

This comment has been minimized.

Copy link

commented Sep 18, 2018

Using btrfs commands I could remove those sub-volumes e.g. :

btrfs subvolume delete eb669bae4f4aa17f3c432d956f481146e4ac77e3f1803fee15e1f2b17787510d-init

Thx devopxy, that work for me to...
This command delete all sub-volumes present is the current directory :
btrfs subvolume delete *

@t3kka

This comment has been minimized.

Copy link

commented Nov 8, 2018

Just ran into this today and had to clean it out. There doesn't appear to be any real solution...is there?

@julienlau

This comment has been minimized.

Copy link

commented Nov 28, 2018

Anyways it seems that btrfs need some periodic cleaning operations.
On my side, I use this function in my .bashrc to do it:

unalias btrfsCleanup 2>/dev/null
btrfsCleanup() {
    echo "btrfsCleanup"
    sudo btrfs fi show
    sudo btrfs fi df /
    sudo btrfs fi usage /
    sudo btrfs balance start -dusage=80 /
    sudo btrfs scrub start -d /
    sleep 120
    sudo btrfs fi df /var
    sudo btrfs fi usage /var
    sudo btrfs balance start -dusage=80 /var
    sudo btrfs scrub start -d /var
    sleep 120
    sudo btrfs fi df /var
    sudo btrfs fi usage /var
    echo "Done"
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.