New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker gradually exhausts disk space on BTRFS #27653

Open
JonasT opened this Issue Oct 22, 2016 · 40 comments

Comments

Projects
None yet
@JonasT
Copy link

JonasT commented Oct 22, 2016

Description

Steps to reproduce the issue:

  1. Install docker on a btrfs system
  2. Use docker intensively for a while, and make sure to remove & recreate containers, rebuild with no cache, restart the system, ...
  3. Check disk space

Describe the results you received:

  • /var/lib/docker uses up more than 30GB, with all except a few megabytes located inside /var/lib/docker/btrfs

  • docker ps -s shows a few gigabtes, below 10GB

  • The following do absolutely nothing to change the /var/lib/docker disk space use

    1. docker rmi $(docker images -aq)
    2. docker volume rm on every line of docker volume ls -qf dangling=true
    3. removing all files matching /var/lib/docker/*/*-json.log
    4. docker rm on every stopped container listed in docker ps -a

    If there is an obvious cleanup procedure that I missed, it's probably not a bug but my own mistake.
    I have been known to wonder why disk space runs out so quickly with docker logs growing to gigabytes, without realizing what was going on 😄

  • The following solves the situation (but it shouldn't be required):

    1. <shutdown all services>
    2. service docker stop
    3. apt-get remove --purge docker-engine
    4. rm -rf /var/lib/docker/
    5. apt-get install -y docker-engine
    6. service docker start
    7. <restart all services>

    After this procedure, disk usage of /var/lib/docker goes back to a few single digit gigabytes as expected

Describe the results you expected:
/var/lib/docker is, after the failing cleanup procedures described above, not significantly larger than what is shown in docker ps -s

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

root@Ubuntu-1510-wily-64-minimal ~ # docker version
Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64
root@Ubuntu-1510-wily-64-minimal ~ #

Output of docker info:
(this is after purging and reinstalling as described above)

root@Ubuntu-1510-wily-64-minimal ~ # docker info
Containers: 14
 Running: 14
 Paused: 0
 Stopped: 0
Images: 41
Server Version: 1.12.2
Storage Driver: btrfs
 Build Version: Btrfs v4.4
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-45-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: Ubuntu-1510-wily-64-minimal
ID: 2ILP:SBMD:2NB7:JY7W:YC3K:CYYV:AX5E:FZLB:CON4:3JCT:3GT4:W4PB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8
root@Ubuntu-1510-wily-64-minimal ~ #

Additional environment details (AWS, VirtualBox, physical, etc.):
KVM virtual server hosted by hetzner, operating system is Ubuntu 16.04 LTS (the 15.10 occurances above are simply unchanged terminal prompts, the system has been upgraded a while ago)

@JonasT

This comment has been minimized.

Copy link

JonasT commented Oct 22, 2016

It took a while to run full too, like 1-3 months for 20GB excess - so it's not any rapid or obvious process.
Also 1.12 is the version after using the process above to fix it which involves a reinstall - I might have been on a slightly older version (at least 1.10) before.

@justincormack

This comment has been minimized.

Copy link
Contributor

justincormack commented Oct 24, 2016

Can you provide more details about which files are taking up the space, and docker info when there is a problem not when there is not please.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Oct 24, 2016

Generally docker images would be taking up more space than what you'd see in docker ps -s.
Also volumes.

@CoRfr

This comment has been minimized.

Copy link

CoRfr commented Jan 27, 2017

I'm facing the same issue with 1.12.3.

With a full clean-up:

# Containers
$ docker ps -aq | xargs docker rm --volumes
...
$ docker ps -aq | wc -l
0
# Images
$ docker images -q | xargs docker rmi
...
$ docker images -aq | wc -l
0
# Volumes
$ docker volume ls -q | xargs docker volume rm
...
Error response from daemon: Conflict: remove cae42d591f87461e454f6c1e9489fa4df3be2ea4cffc241276aa51b2029c3649: volume is in use - [41f083c055837846206e561b9dadc81383a86ec3be39ef2683dd488fa102fe63]
...
$ docker volume ls -q | wc -l
397

It seems there is an issue as I cannot remove any of the volume at this point

$ sudo systemctl restart docker
$ docker volume ls -q | xargs docker volume rm
...
# Works okay, 
$ docker volume ls -q | wc -l
0
$ df -h | grep docker
/dev/sda3                      100G   17M   99G   1% /var/lib/docker
$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.3
Storage Driver: btrfs
 Build Version: Btrfs v4.2.2
 Library Version: 101
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.7.0-coreos-r1
Operating System: CoreOS 1122.3.0 (MoreOS)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.38 GiB
Name: <hostname>
ID: KTOO:PHLQ:52NC:RAOF:S6RS:L3PP:LJPW:JD5W:PGVI:3ITE:CHFT:TDX4

There seems to be a leak the usage counting that results in volumes that should be removed but are not.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Jan 27, 2017

@CoRfr Have you re-balanced your btrfs partition?
btrfs requires hand-holding.

@JonasT

This comment has been minimized.

Copy link

JonasT commented Jan 28, 2017

Incidentally, it seems my BTRFS is again back in a state where it's close to running full due to docker after a few months of use.

If you want to suggest commands to run for interesting information (preferably a lot more concrete than "Can you provide more details about which files are taking up space") then now would be the time, since I'll need to completely wipe and reinstall docker very soon again to fix this.

Edit: also yes, I'm running balance and various clean up commands (the obvious old image and container and dangling volumes and logs deletions) right now too, so I'll report back if that solves anything for me or if it stays full as it did last time.

@JonasT

This comment has been minimized.

Copy link

JonasT commented Jan 28, 2017

(all of the following output is collected after a completed btrfs filesystem balance)
Again same situation as last time: /var/lib/docker takes up insane amount of space, but the folder /srv where my all of my actual volumes are (I exclusively use host-mounted volumes) barely takes up 20GB:

root@Ubuntu-1604-xenial-64-minimal ~ # btrfs fi show
Label: none  uuid: fc92f1ac-ee98-4991-8a92-93451bd36d56
	Total devices 1 FS bytes used 46.02GiB
	devid    1 size 90.37GiB used 49.07GiB path /dev/sda3

root@Ubuntu-1604-xenial-64-minimal ~ # btrfs filesystem df /
Data, single: total=46.01GiB, used=45.24GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.50GiB, used=802.70MiB
GlobalReserve, single: total=126.28MiB, used=0.00B
root@Ubuntu-1604-xenial-64-minimal ~ # docker ps -s
CONTAINER ID        IMAGE                        COMMAND                   CREATED             STATUS              PORTS                                                                NAMES                          SIZE
2dc2c808cd04        jwilder/docker-gen           "/bin/sh -c ' rm -rf "    7 days ago          Up 7 days                                                                                proxy_dockergen_1              0 B (virtual 17.21 MB)
fa3ccbbb65fd        proxy_proxy                  "/tmp/launch.py"          7 days ago          Up 7 days           0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                             proxy_proxy_1                  14.34 MB (virtual 376.2 MB)
86d971bbec3b        gitmirror_main               "/bin/sh -c 'bash -c "    3 weeks ago         Up 3 weeks                                                                               gitmirror_main_1               74.85 MB (virtual 599.1 MB)
b0780588506d        gitlab_gitlab                "/bin/sh -c /launch.s"    4 weeks ago         Up 4 weeks          80/tcp, 443/tcp, 0.0.0.0:22222->22/tcp                               gitlab_gitlab_1                783.5 kB (virtual 1.347 GB)
90e7f35704a3        letsencrypt_letsencrypt      "/bin/sh -c 'service "    5 weeks ago         Up 5 weeks          8000/tcp                                                             letsencrypt_letsencrypt_1      3.667 MB (virtual 653.9 MB)
7be334b076cb        nginx_nginx                  "/bin/sh -c 'echo \"qp"   5 weeks ago         Up 5 weeks          0.0.0.0:20-21->20-21/tcp, 0.0.0.0:1200-1205->1200-1205/tcp, 80/tcp   nginx_nginx_1                  1.574 MB (virtual 310.7 MB)
6af6f7c82926        aptcacherng_main             "/sbin/entrypoint.sh "    5 weeks ago         Up 5 weeks          3142/tcp                                                             aptcacherng_main_1             0 B (virtual 199.4 MB)
ea09d05615f5        mail_mail                    "/bin/sh -c 'python3 "    5 weeks ago         Up 5 weeks          0.0.0.0:25->25/tcp, 0.0.0.0:587->587/tcp, 8000/tcp                   mail_mail_1                    38.45 MB (virtual 1.153 GB)
25769e2cdb4f        icecast2_icecast2            "/bin/sh -c 'service "    5 weeks ago         Up 5 weeks          0.0.0.0:8000-8001->8000-8001/tcp                                     icecast2_icecast2_1            63.86 kB (virtual 513.8 MB)
bc245dc9b02b        97e7720dbaed                 "/bin/sh -c 'sh /laun"    8 weeks ago         Up 8 weeks          80/tcp, 443/tcp                                                      informationnode_nginx_1        2 B (virtual 355.2 MB)
0e8404c5ac0e        mariadb                      "docker-entrypoint.sh"    8 weeks ago         Up 8 weeks          3306/tcp                                                             icinga2frontend_mariadb_1      2 B (virtual 389.9 MB)
80fa98d439bb        8183a6366281                 "/bin/sh -c 'echo \"ir"   8 weeks ago         Up 8 weeks          0.0.0.0:2222->22/tcp                                                 chat_irssi_1                   4.48 MB (virtual 366.3 MB)
4dbf65a5e8f2        wobbleninjabackend_backend   "/bin/sh -c 'cd /home"    8 weeks ago         Up 8 weeks          8001/tcp                                                             wobbleninjabackend_backend_1   0 B (virtual 515.2 MB)
5d761c7564bf        b9d8ec10414a                 "/bin/sh -c 'chown 10"    8 weeks ago         Up 8 weeks          0.0.0.0:64738->64738/tcp                                             mumble_mumble_1                0 B (virtual 306.1 MB)
root@Ubuntu-1604-xenial-64-minimal ~ # du -chd 1 /var/lib/docker/*
105G	/var/lib/docker/btrfs/subvolumes
105G	/var/lib/docker/btrfs
32K	/var/lib/docker/containers/5d761c7564bf110bb7d7f23dfa54da575648a7f9a0893a25a21473c8d8c5c62d
28K	/var/lib/docker/containers/4dbf65a5e8f24e7178d3d6625e71021ac5a2f752e48b64279e4c39fc110e746b
28K	/var/lib/docker/containers/80fa98d439bb0d53fea62bd6a273b222e9c453139b3bae7c2b9805129ad6fa40
28K	/var/lib/docker/containers/0e8404c5ac0ec768dae1527d0a3827779c46a2c0464ed31442d9cec3c22c4f69
28K	/var/lib/docker/containers/bc245dc9b02b83721286351dd38232404004575826f0ec2a67e80f1c0d991d0a
28K	/var/lib/docker/containers/25769e2cdb4f5d586389db901290783974d00eb65c1b1d151a436a6ba705ed58
52K	/var/lib/docker/containers/ea09d05615f5ba031897feda09c1fe097303b855b2cb2563a1fbb89e7314604c
28K	/var/lib/docker/containers/6af6f7c829264625f231208f4f9b253ffb7438bfbe4bc8be3f40d3c8c8e32518
32K	/var/lib/docker/containers/7be334b076cb691e8b06a2ff9b542f6df83d40a04cf6515121156dfd31cded7b
72K	/var/lib/docker/containers/90e7f35704a3803007b695194edf775ab0c9c27fa8c284661e29c25c2f1c9635
16G	/var/lib/docker/containers/b0780588506d56093f8027b7acfea72068012c43ad6fef6fd12fa8064965c624
445M	/var/lib/docker/containers/86d971bbec3bcf37d3adf091abd85253a63f2b5ae8e4d55b889dcfc12272dd53
32K	/var/lib/docker/containers/fa3ccbbb65fd16770992c4bf0b042f04e7129c40fe246e614006a70732426626
2.8M	/var/lib/docker/containers/2dc2c808cd04036563b05bc9ae1a5a3e109c80455120c91c5c662057c91d6d30
16G	/var/lib/docker/containers
27M	/var/lib/docker/image/btrfs
27M	/var/lib/docker/image
144K	/var/lib/docker/network/files
144K	/var/lib/docker/network
0	/var/lib/docker/swarm
0	/var/lib/docker/tmp
0	/var/lib/docker/trust
0	/var/lib/docker/volumes/32956ae03e153e19d227521698eb311bf1bd6858c1cb7b6e3e28b16c9fcfe92a
196K	/var/lib/docker/volumes
121G	total
root@Ubuntu-1604-xenial-64-minimal ~ # du -chd 1 /srv/
12K	/srv/.rsync-helpers
120K	/srv/apt-cacher-ng
2.5G	/srv/chat
116M	/srv/dux
7.5G	/srv/gitlab
508K	/srv/gitmirror
88K	/srv/icecast2
164K	/srv/icinga2-core
115M	/srv/icinga2-frontend
3.4M	/srv/information-node
1.1M	/srv/letsencrypt
12M	/srv/mail
156K	/srv/mumble
7.3G	/srv/nginx
136K	/srv/proxy
0	/srv/rancher
1.9M	/srv/unused-services
669M	/srv/wobbleninja-backend
4.0K	/srv/network
19G	/srv/
19G	total
root@Ubuntu-1604-xenial-64-minimal ~ #

I already cleaned up dangling volumes, unused images and stopped containers.
Obviously, something in /var/lib/btrfs isn't being properly removed though.

@JonasT

This comment has been minimized.

Copy link

JonasT commented Jan 28, 2017

root@Ubuntu-1604-xenial-64-minimal ~ # du -chd 1 /
0	/proc
0	/dev
0	/sys
142M	/boot
13M	/bin
6.3M	/etc
0	/home
784M	/lib
4.0K	/lib64
0	/lost+found
0	/media
0	/mnt
0	/opt
68M	/root
53M	/run
12M	/sbin
19G	/srv
476M	/tmp
1.1G	/usr
122G	/var
143G	/
143G	total
root@Ubuntu-1604-xenial-64-minimal ~ #

(the excess over 100GB is probably shared data between multiple volume / btrfs snapshots - the disk itself is 100GB in size)

Also, here a list of all images:

root@Ubuntu-1604-xenial-64-minimal ~ # docker images
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
gitmirror_main               latest              09f724c7c3e5        3 weeks ago         524.3 MB
mail_mail                    latest              9d679356338d        5 weeks ago         1.114 GB
gitlab_gitlab                latest              b4bf5ba6ed8d        5 weeks ago         1.346 GB
nginx_nginx                  latest              b2b76ece6c18        5 weeks ago         309.1 MB
letsencrypt_letsencrypt      latest              41e3fff3cb98        5 weeks ago         650.2 MB
icecast2_icecast2            latest              99ea6bb18bac        5 weeks ago         513.7 MB
proxy_proxy                  latest              6e5bb262c014        5 weeks ago         361.8 MB
ubuntu                       16.04               104bec311bcd        6 weeks ago         128.9 MB
ubuntu                       latest              104bec311bcd        6 weeks ago         128.9 MB
gitlab/gitlab-ce             latest              119b6ca37943        6 weeks ago         1.268 GB
debian                       jessie              19134a8202e7        6 weeks ago         123 MB
aptcacherng_main             latest              972be92584c2        8 weeks ago         199.4 MB
sameersbn/apt-cacher-ng      latest              972be92584c2        8 weeks ago         199.4 MB
wobbleninjabackend_backend   latest              898192175027        9 weeks ago         515.2 MB
<none>                       <none>              b68db382b24d        11 weeks ago        338.3 MB
<none>                       <none>              42c3476fe205        11 weeks ago        542.9 MB
<none>                       <none>              8183a6366281        11 weeks ago        361.8 MB
<none>                       <none>              97e7720dbaed        11 weeks ago        355.2 MB
<none>                       <none>              b9d8ec10414a        11 weeks ago        306.1 MB
mariadb                      latest              66498efd6bd8        11 weeks ago        389.9 MB
jwilder/docker-gen           latest              d24146176a5d        11 weeks ago        17.21 MB
debian                       <none>              73e72bf822ca        11 weeks ago        123 MB
sameersbn/apt-cacher-ng      <none>              d6cc7ed41d72        3 months ago        199.4 MB
ubuntu                       <none>              f753707788c5        3 months ago        127.1 MB
root@Ubuntu-1604-xenial-64-minimal ~ #

Also, as you can see I have no undeleted stopped containers left:

               mumble_mumble_1
root@Ubuntu-1604-xenial-64-minimal ~ # docker ps -a | wc -l
15
root@Ubuntu-1604-xenial-64-minimal ~ # docker ps | wc -l
15
root@Ubuntu-1604-xenial-64-minimal ~ #

And the only volume listed in docker volume appears to be empty on disk:

root@Ubuntu-1604-xenial-64-minimal ~ # docker volume

Usage:	docker volume COMMAND

Manage Docker volumes

Options:
      --help   Print usage

Commands:
  create      Create a volume
  inspect     Display detailed information on one or more volumes
  ls          List volumes
  rm          Remove one or more volumes

Run 'docker volume COMMAND --help' for more information on a command.
root@Ubuntu-1604-xenial-64-minimal ~ # docker volume inspect 32956ae03e153e19d227521698eb311bf1bd6858c1cb7b6e3e28b16c9fcfe92a
[
    {
        "Name": "32956ae03e153e19d227521698eb311bf1bd6858c1cb7b6e3e28b16c9fcfe92a",
        "Driver": "local",
        "Mountpoint": "/var/lib/docker/volumes/32956ae03e153e19d227521698eb311bf1bd6858c1cb7b6e3e28b16c9fcfe92a/_data",
        "Labels": null,
        "Scope": "local"
    }
]
root@Ubuntu-1604-xenial-64-minimal ~ # du -ch /var/lib/docker/volumes/32956ae03e153e19d227521698eb311bf1bd6858c1cb7b6e3e28b16c9fcfe92a/_data
0	/var/lib/docker/volumes/32956ae03e153e19d227521698eb311bf1bd6858c1cb7b6e3e28b16c9fcfe92a/_data
0	total
root@Ubuntu-1604-xenial-64-minimal ~ #
@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jan 28, 2017

Would it be worth checking if there's a container that writes a lot of changes to the containers filesystem (i.e. not in a volume), (you can docker diff <container> to see files added/modified in the containers writable layer). Any particular btrfs subvolume that stands out?

@JonasT

This comment has been minimized.

Copy link

JonasT commented Jan 28, 2017

@thaJeztah but shouldn't that show up in docker ps -s? As you can see above, there is no significant storage reported for any of the containers in docker ps -s.

@JonasT

This comment has been minimized.

Copy link

JonasT commented Jan 28, 2017

I just tried docker diff <container> on most of the containers. There is nothing immediately suspicious, but e.g. one container checks out a repository (which I know is below 80MB) so it is quite lengthy and I cannot tell you for sure how much that actually changes since there is no size information in the output.

@JonasT

This comment has been minimized.

Copy link

JonasT commented Jan 28, 2017

Output of du -chd 1 /var/lib/docker/btrfs/subvolumes/*:
all-subvolumes-duchd1.txt

Output of btrfs subvolume list /:
btrfs-subvolume-list.txt

@JonasT

This comment has been minimized.

Copy link

JonasT commented Jan 28, 2017

I also collected the docker inspect output of all containers, but I cannot upload it here publicly since I am not sufficiently certain that I managed to remove all confidential information (access tokens, ..). However, if some docker contributor wants to investigate this, I'd be happy to give them access to the file through a private message.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Jan 28, 2017

@JonasT Thanks, I'm going to dig into this further and hopefully have a tool give you to help analyze this without leaking confidential details.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Feb 20, 2017

ping @cpuguy83 have you had time to look for that tool?

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Feb 21, 2017

Not yet :(
But I did create #31012

@huegelc

This comment has been minimized.

Copy link

huegelc commented Sep 21, 2017

any news? just hit this bug

@JonasT

This comment has been minimized.

Copy link

JonasT commented Sep 21, 2017

Yea this is still happening for me as well. @huegelc what I do is every 1-2 months I stop all containers, uninstall docker, rm -rf /var/lib/docker and then reinstall & rebuild everything again and then I'm fine for the next 1-2 months. It's not the prettiest solution, but if you have a fallback server or you can afford 30 minutes of downtime every 1-2 months it works.

@huegelc

This comment has been minimized.

Copy link

huegelc commented Sep 22, 2017

Thanks. Well, using docker with btrfs isn´t usable in a production environment yet as of the uncontrolled growth of volumes. Really looking forward to have this fixed.

@johnharris85

This comment has been minimized.

Copy link
Contributor

johnharris85 commented Oct 26, 2017

Ping @cpuguy83 @thaJeztah, as btrfs is the only supported filesystem for SLES, this is an issue for corporate EE customers. Any additional updates / information / workarounds you'd recommend?

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Oct 31, 2017

Are these issues occurring on a recent version of docker (ie. one that includes #31012)?
It's definitely curious that there don't appear to be any subvolumes in /var/lib/docker/btrfs/subvolumes... does btrfs subvolume list /var/lib/docker/btrfs/subvolumes show anything?

@johnharris85 Docker EE customers should report issues to Docker support.

@hopeseekr

This comment has been minimized.

Copy link

hopeseekr commented Feb 3, 2018

I've created a comprehensive guide on how to get yourself out of this quagmire: https://gist.github.com/hopeseekr/cd2058e71d01deca5bae9f4e5a555440

@JonasT

This comment has been minimized.

Copy link

JonasT commented Feb 3, 2018

For what it's worth, one of the quickest ways to solve it is still 1. shutdown docker and all containers, 2. uninstall docker, 3. rm -rf /var/lib/docker, 4. reinstall docker, 5. rebuild & restart all containers.

That gives you some downtime of course (depending mostly on how much you need to rebuild), but it worked fine for me as a comprehensive "reset" without more drastic steps like reformatting the entire disk.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Feb 5, 2018

@hopeseekr, btrfs subvolume delete is exactly what docker does to clean up subvolumes, so I'd be curious as to why it's not cleaning up correctly.

#36047 might solve it?
Not sure what happens with btrfs subvolumes when they are mounted in another namespace since they aren't really ever mounted to begin with.
If someone wants to play around with this it would be helpful.

@hopeseekr

This comment has been minimized.

Copy link

hopeseekr commented Mar 7, 2018

I honestly think that Docker / Moby (The singer should totally sue you guys!) loses track of BTRFS subvolumes under certain circumstances. It appears to me that they are recursively nested at times, and that's when the full file system corruption usually occurs.

This is like a memory leak, but instead, it's a disk space leak, which at a certain threshold makes the entire file system seize up and die. You end up with a frozen system that spontaneously reboots or enters into read-only mode if it's not / and then you get nasty kernel messages on the next boot.

@beerendlauwers

This comment has been minimized.

Copy link

beerendlauwers commented Mar 21, 2018

@JonasT's solution did not work for me, unfortunately. When building images, the build always failed with a No such file or directory error of a subvolume in the /var/lib/docker/btrfs/subvolumes folder, even though that directory had been cleared with the relevant btrfs subvolume delete and rm -rf commands.

@hopeseekr's workaround did work for me.

@JonasT

This comment has been minimized.

Copy link

JonasT commented Mar 21, 2018

You need to remove the entire /var/lib/docker folder and all subvolumes inside, not just some part of the inner contents (so everything is truly gone) - that always worked for me.

@beerendlauwers

This comment has been minimized.

Copy link

beerendlauwers commented Mar 21, 2018

@JonasT I did that, I first uninstalled Docker, then nuked the subvolumes with btrfs subvolume delete, then deleted the /var/lib/docker folder itself, rebooted, and then reinstalled Docker. My btrfs is probably further gone than yours 😄

@JonasT

This comment has been minimized.

Copy link

JonasT commented Mar 21, 2018

@beerendlauwers that sounds exactly like what I did (minus reboot). then you're getting errors building an image? that seems odd, there should be nothing left around if you do that procedure. but I'm not a btrfs expert, so I'm afraid I can't tell why that would be happening.

However, if you truly haven't left anything over, that kinda sounds like a btrfs / kernel bug then... but again, I'm the wrong person to make an educated guess here.

I think I read somewhere though that btrfs subvolumes need a bit to be deleted, but I might be misremembering. Maybe you would have needed to just wait a couple of minutes longer? (without a reboot) A btrfs expert might know more on this.

@stefangweichinger

This comment has been minimized.

Copy link

stefangweichinger commented Sep 1, 2018

This bug is old but still open ... so I add my "me too" here in august 2018. I also have these issues with docker on btrfs, with Fedora 28 and the docker-ce.repo (test setup on my desktop system).

Removing /var/lib/docker every few days can't be the solution for software used for infrastructure IMO. I would appreciate any hints or tips on this issue.

@beerendlauwers

This comment has been minimized.

Copy link

beerendlauwers commented Sep 3, 2018

@stefangweichinger

This comment has been minimized.

Copy link

stefangweichinger commented Sep 3, 2018

@beerendlauwers thanks, yes, read that. so "workaround instead of solution" so far.
Interesting: on one machine docker uses "Storage Driver: btrfs" while on another it uses "overlay" as driver ... both machines on btrfs. But this might get off topic here, more of a Fedora specific issue. I also wonder if the howto by @hopeseekr shouldn't also mention editing the daemon.json to change the storage driver in case it was on btrfs. Or does docker detect that at starting on the pseudo fs?
thanks all.

@danbeck

This comment has been minimized.

Copy link

danbeck commented Oct 31, 2018

I got the problem, too. It's quite annoying on a productive system, where I cannot simply uninstall docker, remove everything that is docker related and reinstall.

Isn't it possible to find out which BTRFS subvolumes are unused by containers and images?

@funkyfuture

This comment has been minimized.

Copy link

funkyfuture commented Oct 31, 2018

those who are still reporting on this issue should mention the Docker version they are using. on Arch Linux, which always has a recent version available, this behaviour stopped a while ago.

@danbeck

This comment has been minimized.

Copy link

danbeck commented Nov 5, 2018

Like said in #27653 (comment), btrfs subvolume is growing.

Used distribution: Ubuntu 16.04.5 LTS
Used Kernel: Linux kinext-da-2 4.4.0-134-generic #160-Ubuntu SMP Wed Aug 15 14:58:00 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

docker version client: 17.12.1-ce
docker version server: 17.12.1-ce

The size of the btrfs subvolume-directory is:

114G	btrfs
16K	builder
1.4M	containerd
36G	containers
26M	image
212K	network
0	plugins
0	runtimes
0	swarm
0	tmp
0	trust
19G	volumes

docker system df reports the following information:

docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              30                  25                  9.887GB             3.444GB (34%)
Containers          26                  26                  10.36GB             0B (0%)
Local Volumes       23                  13                  5.159GB             0B (0%)
Build Cache                                                 0B                  0B
@H4ndl3

This comment has been minimized.

Copy link

H4ndl3 commented Jan 4, 2019

I too encountered this situation. It seems that the folder size of /var/lib/docker/btrfs is misrepresented by the usual file system tools.

In my case it looks as follows:

du -sh /var/lib/docker/btrfs
30G /var/lib/docker/btrfs

btrfs fi usage /btrfs (/btrfs is the mount point of my top level subvolume)

Overall:
    Device size:		 745.75GiB
    Device allocated:		   5.02GiB
    Device unallocated:		 740.73GiB
    Device missing:		     0.00B
    Used:			   3.66GiB
    Free (estimated):		 741.26GiB	(min: 741.26GiB)
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		  16.00MiB	(used: 0.00B)

Data,single: Size:4.01GiB, Used:3.48GiB
   /dev/mapper/root	   4.01GiB

Metadata,single: Size:1.01GiB, Used:183.33MiB
   /dev/mapper/root	   1.01GiB

System,single: Size:4.00MiB, Used:16.00KiB
   /dev/mapper/root	   4.00MiB

Unallocated:
   /dev/mapper/root	 740.73GiB

@danbeck can you check if that is the case for you too?

@p-na

This comment has been minimized.

Copy link

p-na commented Jan 17, 2019

I think I've got the same issue.

home btrfs # pwd
/var/lib/docker/btrfs

home btrfs # du -hs subvolumes 
140G	subvolumes

home btrfs # btrfs fi du -s subvolumes
     Total   Exclusive  Set shared  Filename
 127.98GiB   966.06MiB     4.84GiB  subvolumes

home btrfs # docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              20                  1                   9.64GB              7.675GB (79%)
Containers          1                   1                   170.4MB             0B (0%)
Local Volumes       13                  2                   55.61GB             37.51GB (67%)
Build Cache         0                   0                   0B                  0B

home btrfs # docker image prune -f    
Total reclaimed space: 0B

home btrfs # docker container prune -f
Total reclaimed space: 0B

home btrfs # docker version
Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:31 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:02:44 2019
  OS/Arch:          linux/amd64
  Experimental:     false

home btrfs # 

Files in /var/lib/docker/btrfs/subvolumes:

home btrfs # du -sh subvolumes/* | sort -h
1,2M	subvolumes/55dd00a7755afbdd68b00a490906d53f26ecc68cdd8a61418f04385d4e944001
4,3M	subvolumes/55a4d5f0a00070e88e0bfb8e6079b6281b736e67e0249a4cc43fc8e8e28a99bd
5,7M	subvolumes/0ffe0f7d7b2d08e1f9e2d1594d58a7fb7221db939efcbb8e85a9dfcf6e44f096
5,7M	subvolumes/99214ed9a6138357850e8abf68df7c6582fe2d8d0162e63945a2b16e94f44e55
6,5M	subvolumes/6886f7dee296d644b35ee9ec4694ac5b2c02df86d170c43ec2d5d1667b5183c4
8,6M	subvolumes/d2e31574a87ef483eb547ed79d66e9c0a5c7263d49b6813ab9c00ef2ce631875
9,0M	subvolumes/40f785ff282f86239060981bac80bd1b2b5cabcec284db66a5a7edb34337b36a
12M	subvolumes/554a46401a7e8bbda2b3a309e4e5309c13f4863784b4c3e548979bb8e261953b
26M	subvolumes/6caa37ab9d2590093d90663a944f9110ec32fcc228ccc9d7718a88820c5e7018
43M	subvolumes/79643abb242fb439ed45bdadce0440ba3a4a2af17f70d59ae45f254d6c33291d
43M	subvolumes/b61e63ca772a95c18ae3da261327ed56980858a23577616082c03d9896d861ff
43M	subvolumes/d85e678e3b340dbab58b8689b20f979e23645dca97e66449ad0358c9a4012129
64M	subvolumes/285b39dbc2228befa9a57d8892a9a16579e0afe76809d17845295d3562d74661
65M	subvolumes/6e9eeb88c6d2a0dc26652afe6a06b73f1bed19ec22923bc879780067c593bf68
65M	subvolumes/869b157629cee8d4ea467692bfcce527a3ea6635898d2f05aac77c338585e657
74M	subvolumes/0c8b1674c17d4e3e14090ecae8bee108c4449ef52f6928eb268d387227fd8d0a
74M	subvolumes/69eb9d9e7ffeff5e1ee3bb86760929ea27d5420e32061b47d7aa12acc40bf811
74M	subvolumes/a7ec6f11e38f08217dbe726b197f94f8c5c2b024c29cb44fda514d1a655b36c4
74M	subvolumes/f1bca1ffc91f11f136838dc869787247ee05519ef60e49268b56c65edad243ae
85M	subvolumes/1bf5c9a16ef3272411bfd95cc31fe436ee90446797fc36a5d729fb256cc109ad
87M	subvolumes/18860bcd4173e2bcfc7fef820749c0a9f9848fca8292b1b28b5d435ec5276a1c
87M	subvolumes/ab219e5ccbd160ff6812c01dd9a19fd2f801ddb01a278863d96aee92a93c73c4
90M	subvolumes/449d017aaf7dcd26321d4d0b1b2ff4c63028138798310a98f71686989c2a76aa
90M	subvolumes/582ca8c0a7c5bbdef70222a261c86092310b0894b3b2296d14f7a629fbe446fb
90M	subvolumes/82794b057109b226ea8bc6ffdc549009adcaba1a4b8aa410aceb232988cb0e6a
100M	subvolumes/8e58d863c19fb22fc7a192359cf1bf656f2efbe92bb2066d82397f1cecaf08cb
100M	subvolumes/9754a36619f44d8b295857aabccebc86c24230465696d2978e1ec840947cc019
100M	subvolumes/c1381438843c335e451f67fdcf3ab45e1a528d62ba854494b81ce30fac6995b5
100M	subvolumes/c7db7a2a4d63118c6435b9a6c88a71d4c9a651b403f2b225cecdf9f13ccca8d7
100M	subvolumes/cf7002597a965d1ccc174be99565948adf05507efd7372628f7e6559d013ba80
101M	subvolumes/218f673988186715b1555baf90322419dbf4835b53ee0618bd1459dc078d9aa3
101M	subvolumes/56c08ee93f1ff9a610ac723be6d82c252b4fdbb21b987b055a6c0252a35c5d57
101M	subvolumes/b7e792b3329ca71ef7ca4927d2be31748abc06806c3f9b96f687a33c21dddbef
106M	subvolumes/13ac22fa33b8f12cef52f4419fe881ec70e136abeb1af7a4f9c232ca2dc112ca
109M	subvolumes/31289a5cf8faeee120d11b770f417b59030dfa04b04396c210023e0f32a7b163
115M	subvolumes/0df6bb601620ad96c7272e7ebae3b235d563bf07cf6ca52343e8b30a0469b12d
116M	subvolumes/bef0fae560dca1e28def65b24e1011a8b67f18c93b59cd6331414eac957d05e9
140M	subvolumes/7b0c54df2bcc1517ec46d1dddbb657453b9412a899b6bc9d2c89a8ce14449958
140M	subvolumes/f404991de873e6887b99e6f9a031347af40c8b778004241bde573d6a9deda3e0
149M	subvolumes/9764f67ade1e50c4c5578241a0d1893f922f63afef4640e507b2d71425cc49fb
151M	subvolumes/29d287d16b0ecfa81a3d37006971b86d146c69d953afb9ed50fc61c9438f0f1e
151M	subvolumes/45de4716607c29cdb2b7aeb094ca6e9fb3905c0e232e53690ce1f6b21ce76132
209M	subvolumes/586e82953ea9efb8720a03e1d1d3ae98e8dfe535fd05a93c35e1f668bde5250e
209M	subvolumes/594378c21fd126fcc2a666621ac89a631dd8d0952c069242a7399d2e462b4416
244M	subvolumes/0dd2ce750b854af86acd329233fd3a0015f2e16152d72c264250091bdf9c04f8
244M	subvolumes/4b78b1fabd55f5a945a3b43f68050c153d71f1209e3385928a35e6eb95325dca
244M	subvolumes/9ada3d76913727f14e3430eddfd392220fc02495ac3f556555f66cb8acb1bc5c
266M	subvolumes/9e22a1d24e2ff2473ea4c8f17683bb27fefebc728cd327d308e0c94c6c51f829
267M	subvolumes/f674e2c2d1e4f2c74b23757b3d5026f19cec4160503c6c72be77577139df4788
277M	subvolumes/41cb81803b34136696b8c57c2600a6e7b77fe15e23fb933c02c049d3eb42bc58
279M	subvolumes/a6c7c0a28b63d9bfd91c3d9c224226ecf4e641f752943eb0802219fd3bea91bd
292M	subvolumes/7cd27d3d4f441948c744d72253cf2e506c32a2305e03a7eda8be35f9baba71f3
292M	subvolumes/d1e4029cdef899d4a3ca3815d646c047dc19ec15f7d45d2e496a970c7f91d029
306M	subvolumes/d9bb30c2ff8b972a9407e4d1fe7ce0897e5aa918ea5e7b1484f798824a8a901f
306M	subvolumes/ea736443494a119f1c2ec3e9c03ce9d92f598872c541934774755121b69abcee
350M	subvolumes/0d03db536837d36818b5c9c0575555631a4edaf9df9600da239aba4b77e6b3c5
350M	subvolumes/4f39995f6f23a9e5a1e8e46c823068bb48dba370788a66821cf0b4b191933426
350M	subvolumes/aa8bab5ef4ae504743ef5de6a68aa94947207264593e72987fc972c7fefdbd90
435M	subvolumes/5b29a7ff99b41cc1eecf67eeb687a08221b17e466f7e37ac6bb1fcf825866f70
438M	subvolumes/bab936785954df8c4d41439ff3a75c96928ddf17e92cac1c58b2b0fe4d547548
443M	subvolumes/0d8345bc1800c1701c96de0462441634946111a65eaad8a1e128b3714e02c66a
458M	subvolumes/472252b83c151db89cd6989a52917b7372a4c709b87001658ef47d87aad1c5a2
466M	subvolumes/b97f4a6a32d418baedc4a2675a375958ed56649389f43f7ea20eb87988393cb3
480M	subvolumes/e6128783b7ad7e1d432ecdd6c4cd0b1f5d83a15a49851dfb6108c62d988fb944
541M	subvolumes/a199845be9af217d6dcdfae6cceaf3dcd5703507ae28e91482bde7fd5c3b53f1
544M	subvolumes/02114fbae74144b9ea6c9e5a6ec6d068f20cbf691eed57c633002070f0b6491e
544M	subvolumes/82524945995f8914900ac2a768ff03ceabfa7b265405af5353d114c5102cce11
544M	subvolumes/90782b900432e2ee026a424bc84dbcc11739ab24e8e0593ba4b035a2a8de9457
544M	subvolumes/f106089a1f87be4100c101a8d306c57396bdcad0897d3f405bdcd7c0daa0f0d9
544M	subvolumes/fbb99cdf699fdb0eeaffe3d6dea9806a817b7eabbfc5c2dd6a57fd628cb7007f
635M	subvolumes/5b52dff33ce148ad9657b80d774bf42d992bf14a602c459eff4fd57f6dadc83e
635M	subvolumes/7b3f2d9ad9d5c32f6e0abae93abf85c25d673b253999060fe6b123ba7b2afe79
635M	subvolumes/cc132a8f901e6b0d452a582d79820314f66db30470f1512c664c1d0345dcb696
709M	subvolumes/3d6be85a285a78fbd3ad8afa873e3ffedf85ad4ed54c02cab8bcc4f4ccf6725c
709M	subvolumes/632dad75958b7e7ba31d306cc595ba5b23055540f28401ff3d39af075a01dfc6
709M	subvolumes/ba007de1c911af0cf5755c51fec3a77aad343c613a163f254624278914736af4
794M	subvolumes/9ced7729f5b111e59550d318453ea5479b54a2dcc458c3b847f8a611fae7a0e3
833M	subvolumes/41a2a3e2142b90982265a56d44823a519c4976f6292bb627797f1b134fd45edd
833M	subvolumes/4529126f3231f35363c5ccc297e855eb28d8c15809541cbced35fc96e5d8c07f
833M	subvolumes/a1831bf833fd99eea84bfe15068c3d338bee455648ec5892d5e7cc45fc56ac3e
833M	subvolumes/c4704b7b397ee6ce84005e15b7711f9dbd0be8577abe16c76bbe86adb840a871
833M	subvolumes/e9204a4523716707d4dd97a31e0156c35b2d74cf7c2600e366be8ac1370b6170
1,1G	subvolumes/2dd0ee4d8a6adce4cf7a7bce33d54e7ae83762a163a684bd22f983bc5600ff9c
1,1G	subvolumes/2e00b4c26d53486fb01f5fa969c079969a96c995d201470244b32b23b2bf9076
1,1G	subvolumes/3e05b80d47f5627c50386ba6e6ec66c1d2e9d8ae4490c25a67d18f6cc923879d
1,1G	subvolumes/45915a7368faae775d9bf931120bb6acc4124b0d9554897b0da1eb7945af997a
1,1G	subvolumes/5557f8fb7699ef044e124fb7a1940c2defdd98df12da40aadbc52117280e995a
1,1G	subvolumes/5c0f3646d08e6f102158134d554a8479dbf13eca7c764a8ffa7533dd609d9bd6
1,1G	subvolumes/5d14b3120e8ec4fcac1f327de52d7a16eb4b1922e8a34215508c90a2982e48dd
1,1G	subvolumes/691120cf3f71a6da5f949e9e93a9a5b7112b5b37de304953d9173382bc23f2c2
1,1G	subvolumes/8be21e0d0f984abb06578720394db5a991c48a5a3d974ac2bc179a527bda948a
1,1G	subvolumes/8df7791f551cf837a34d63e8372b233da0dd4d4faa2c9ca8d9c404b52c1b91b8
1,1G	subvolumes/9c7311720cf3c4a09bc34692de71c722f0abbf53c8c25df89556b3f275d2e523
1,1G	subvolumes/a488b6a9494c4d08f4cd03cb5769134e93b9d2252aa056e3f379e0fc35af41cd
1,1G	subvolumes/b258a17784fb2b4b0def91d8d6e0044750cace85139e48c2681f52051d6990f7
1,1G	subvolumes/b7470f7306b3c5fd39f91dc3ed6f2c5e00de33312fa571744a98d86b2fc70a56
1,1G	subvolumes/bcac7a98c5d745e172b5b6bf447e13ad6f24b912a95e942618d32ed5dae351d0
1,1G	subvolumes/ccc720bcd1c4552ed9982fdc8afa165e5dcb7f7ef2db93d58629f7e55842180a
1,1G	subvolumes/d92d13ab3d3fcf108b5c0c9482345b980fb9cc3c41574865968ca882b2050c67
1,2G	subvolumes/3384226dd0716910e9a92e4810cc878502761644479dd4b0deb2a0eb7a6c3327
1,2G	subvolumes/57197890a33dbabae0c611a40fd14031aae786dbd0c2b0635f2d53bf4e742547
1,2G	subvolumes/5952edf57e9e2b44f982778d297b9f151a60b43e992e2aef4de97d28ec917601
1,2G	subvolumes/8700e09db7d1696f7348de5bbd238265321fb163214d8fdfe2c72fdaaba1cdce
1,2G	subvolumes/876c211c7df97a4e1334915bf437233436790c27d50a5bc9ab9715646e8c28b3
1,2G	subvolumes/abb99b4891fac7e7a732a9a1400c8d6a206be3f6f9bdf6ec9260ee1390cabb82
1,2G	subvolumes/ad45cb0c46a20991e5d8f4e6960e6f76a085cd83bd118fd2f95dc635e73373ae
1,2G	subvolumes/c0b2dfc9b77d5ba8647daa23fa1b258374598f3bcff7e570cbb0c44bc255dabe
1,2G	subvolumes/e80769d058f7ac26bc1d1d764dca08c2837d9cbdab1bf0599d241c43d822929f
1,2G	subvolumes/f8aded1011af82cd9c8ca5ccec8e1648c1cc5784b93f6eba10a9251534ae318b
1,2G	subvolumes/fa0a0c3c9c42467cb73552e5e422cc4136137ca192f0c6211f9768073c0b6bd8
1,2G	subvolumes/fdabea639b205e2e2a4aa9e5e115940cd0a6ea1a683702f0d33715c52262df74
1,3G	subvolumes/0b2342a91807e5b7538917d0d060e6af5274878774a169837ac9748e155af553
1,3G	subvolumes/21d8467a0b339b2b0ea70d2d1f72e7916854ca410429e89453150583f196833c
1,3G	subvolumes/765305d4975ddeb553cca7bd33a62212a625b237299c85dce704a28dfed5cc1a
1,3G	subvolumes/767874f10efffdd072d548ebe302d2fb0976c55d8a11dfb5f8d789bfc2a1accf
1,3G	subvolumes/8b5d714ca1f6259b95dea7750136574318f9d0c500148eceb14e744b5343cd49
1,3G	subvolumes/8ec59864dbb87234dc9a57af1e4641aaf425c864e4e9fd0b2909b4f7d7f8a855
1,3G	subvolumes/93c5ae097872d7b05a542762271520b0432413d21eb8ff626e121cdcb9f887e3
1,3G	subvolumes/a45aed155c16c1ade2607c8aa9c1ab55dcf745985078015c0728a5867069eccf
1,3G	subvolumes/ac9e5eea029d57f935cfa7f8b27eec6c895f6b7dfaec055d7930e72164c0f06e
1,3G	subvolumes/d3cc60cc0b7e3d1d393ccdc2502c492f45fce4425ce7e7e85fff4e762c07a5d2
1,3G	subvolumes/d7fd8187053fb183eb8698b1df1aefa574b71d9101acd1d2a914fc993d4754c2
1,3G	subvolumes/e14aba62ae82fb6d9f0fa6a980eaba989b210822e38ad65f44c0c9a39f5a8387
1,3G	subvolumes/f2688987e17aea9e241f9b851455422011920693ebcc3069b54c5a4fd47ee4e9
1,6G	subvolumes/a44b8975e4eec2183839a3b450b94d958d36059420c6bdf9330b7833a8a20585
1,6G	subvolumes/df29a90443a2c5bcf273f35296d6d3ef3690d56676700434abcc43f4b966b32c
1,7G	subvolumes/1146d611b3133c06e94265c591786583fdb33b634e33e34054d57c0deb68ec4a
1,7G	subvolumes/201111b5afbb4e526f4083853c267df035140bf634c49ef57712ca9adcdd0789
1,7G	subvolumes/299dd5dc5d502e9b5f10c7e0a7f7c4f6dae60b7175273680466f4cb5e8dd578a
1,7G	subvolumes/385d5cd26284d133e27093d017b0986368863694804ef82082a29e068e645c2e
1,7G	subvolumes/61415d3dddc75017b34123e2e6ed6ebb7543795d8ce12910924a40c4eca3c252
1,7G	subvolumes/6363602b6bec5a1ada65cb6427a9ddd40dca61d9fef4966db879dfbe728c68dd
1,7G	subvolumes/7191ce58b48db55e16f968a157c1bb67c339559555cda1d722f56a293fb52ce4
1,7G	subvolumes/7d858ccaf95082356a622c7a85d0795ea040d201d516c1f02fd5dc93438c2453
1,7G	subvolumes/7f7f00d88a0efa2300b0db8fe3aeacd6f70fbb8b981086493c017fc260d927a9
1,7G	subvolumes/831d9319858e9b44ffda0d38af3156c6488b828dea97cbffd7983d0e6fb966ed
1,7G	subvolumes/97dde2c417b5c06e245cf62ba702b6bf769b371a76365f6e5cd7102b73360b69
1,7G	subvolumes/9d81cfc9993f09fcccaf9c51fbeab9e13ca84b01a5eb8033f2e94a4bc27615ef
1,7G	subvolumes/bfa86b1b2c057056aa48b8a888eb7715732d903ae41e2b8edfed113c04785ac2
1,7G	subvolumes/ea2c4cacda6dbf7ff3565bede8c828469208ad90038d4249038de3cafcef6b3a
1,8G	subvolumes/0ccfecf366ac4e36ec8a6729cacf21f3df349d3903ad64a67d37df8d59a2bfdd
1,8G	subvolumes/15dc2967268b41644cdb8fe420194d75cfcf823045657eeb72d01d6c6a413b56
1,8G	subvolumes/20fe33cba60b8c530fd3203ead5076cebc1ec244d8c2b707b857923b901bc837
1,8G	subvolumes/5e59fcede8789cd39e7056b97467c3c9087f612959d3ac5c3f51ffbffd82116f
1,8G	subvolumes/9def4c8af29ea46ff48402fd4b84d87de659d8383025d7ca229c09b942e99f4c
1,8G	subvolumes/c0627c9603a2172928303a16d1bf90c4a871c57f7aa0c3def4d62f46f7e0cf38
1,8G	subvolumes/c35b3cafcf3262653b30eb8cfb9c0f4dec4667f7b9c3e67a59ec10420fb283e4
1,8G	subvolumes/d05138acd5b62f4d0e4efec1e98f11552e42b0e1190d72d52ad5b6dd813533e4
1,8G	subvolumes/d34fa34e4a1d0d79d642324c15d9dd5bf5b557030281b2d17740710b18b8b9eb
1,8G	subvolumes/e88b499f48c09de9da7a58b79227692a085271e47c9b889c90890ee96a799a44
1,8G	subvolumes/eb80db94f11bc08e2ec0b965914c2e5c713562697c62c26359bec0c49c6f63f7
1,8G	subvolumes/fa18d5751d17d26be5acba633b4845e5a88d97feeaf691086eb669b05ab3c21d
1,9G	subvolumes/1d325439ded51aa9d8bfe6d0743ba46f3e7f540f9ba3f9726a90b907d1e61230
1,9G	subvolumes/222a70f6d39659f5443d4e847e20f909323906e5126e0cd6dfbb6c6afe07520a
1,9G	subvolumes/5c85fd2a94b4b092645a7cf650e3b0688c21e13368bf2af26c545a9b149b53ff
1,9G	subvolumes/6acfed0459ddb052aa8cebadeb04f2f3adbab3abe0197434e0878b94331b8920
1,9G	subvolumes/76575745e0d7803ac6e5ab7ddebbc110b04a315c20061d62d87de27d360e2e31
1,9G	subvolumes/7f4b243c53cdc88aa6afd7aa114081366be4a7bfec2ec9f090afab8b6178517d
1,9G	subvolumes/7f4b243c53cdc88aa6afd7aa114081366be4a7bfec2ec9f090afab8b6178517d-init
2,0G	subvolumes/2cc223375bebba2101ad6e62ae8f171f985bb0f3c4a3c36351f57d0ccf2d1b45
2,0G	subvolumes/7d4ad44bf4d3204052c17ea3a20f876549fefa0951f8d1f0f092b7a068480309
2,0G	subvolumes/99267f97b0ec6c348ead4916dadd3ca9afd955102f914bf6b3d72bcb5b14e368
2,0G	subvolumes/c797110d4a05c057e4c470e1ea51ee654503c31031d12798ecac97433cabc175
2,0G	subvolumes/defe926b79c58a21dd67d7077bd3573b482a2a6dba3d63a3674c1f09619ab7fe
home btrfs # pwd
/var/lib/docker/btrfs
home btrfs # 

I use Ubuntu 18.04.1 LTS with 4.15.0-43-generic kernel.

Please let me know if I can provide you with any additional information.

@H4ndl3

This comment has been minimized.

Copy link

H4ndl3 commented Jan 18, 2019

@p-na please do a check with btrfs fi usage on the btrfs top level subvolume (subvolid=5).

@p-na

This comment has been minimized.

Copy link

p-na commented Jan 18, 2019

Since I posted the last time, I deduplicated /var/lib/docker/btrfs/subvolumes on a file level and regained ~ 20G. But they're also nearly gone.

➜  ~ sudo btrfs fi usage /
[sudo] password for user: 
Overall:
    Device size:		 194.68GiB
    Device allocated:		 144.03GiB
    Device unallocated:		  50.65GiB
    Device missing:		     0.00B
    Used:			 109.45GiB
    Free (estimated):		  77.63GiB	(min: 77.63GiB)
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		 215.84MiB	(used: 0.00B)

Data,single: Size:133.00GiB, Used:106.01GiB
   /dev/sdc6	 133.00GiB

Metadata,single: Size:11.00GiB, Used:3.43GiB
   /dev/sdc6	  11.00GiB

System,single: Size:32.00MiB, Used:48.00KiB
   /dev/sdc6	  32.00MiB

Unallocated:
   /dev/sdc6	  50.65GiB
➜  ~ 
@H4ndl3

This comment has been minimized.

Copy link

H4ndl3 commented Jan 20, 2019

@p-na the actual disk usage of your /var/lib/docker/btrfs/subvolumes folder as per your previous post would be 966.06 MiB (column Exclusive).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment