Inaccurate Results from docker system df

[braunsonm]

Description

Steps to reproduce the issue:
Unknown, however this results from using multiple FROM statements in a Dockerfile with docker build . in my experience.

Describe the results you received:
After running a build, Docker seems to keep multiple layers in it's VFS storage that cannot be seen through docker system df -v

Note the results from docker system df -v

Images space usage:

REPOSITORY          TAG                 IMAGE ID            CREATED ago         SIZE                SHARED SIZE         UNIQUE SiZE         CONTAINERS
<none>              <none>              dbcf26a1afe3        29 hours ago ago    1.754GB             1.748GB             5.572MB             1
microsoft/dotnet    2-sdk               9969575612df        2 weeks ago ago     1.748GB             1.748GB             0B                  0

Containers space usage:

CONTAINER ID        IMAGE               COMMAND                  LOCAL VOLUMES       SIZE                CREATED ago         STATUS                      NAMES
e81177326b98        dbcf26a1afe3        "/bin/sh -c 'sed -i …"   0                   349MB               29 hours ago ago    Exited (137) 29 hours ago   jovial_babbage

Local Volumes space usage:

VOLUME NAME         LINKS               SIZE

Build cache usage: 0B

Note how it also says build cache usage is at 0B. Now note the output of the VFS directory:

1.8G    /var/lib/docker/vfs/dir/0478c82626f2680e23a285b2d8f97f2a91d0ea9bfc0bf35320307c821956bb7d
1.8G    /var/lib/docker/vfs/dir/1f6bee7f0e35932bed711496b14cb58fbb07017c5a98fcf5893bb924e89c2ea4
288M    /var/lib/docker/vfs/dir/2bea78b30abf2e5137e91d963c1980d90c4a9dd58bc4e019710f646ad13548eb
111M    /var/lib/docker/vfs/dir/416dedbc38925c3370753f742e8b8c326a04d1cbaa5302c42eb8f8f28d49b9aa
628M    /var/lib/docker/vfs/dir/5d437a35347ec637156103ed49639481a3f4608adcf59df66685d4f032d83e7b
319M    /var/lib/docker/vfs/dir/6272847cccecd9865fbf7f401caafdf3db65127c56439760d5a2674fd0c6a7da
141M    /var/lib/docker/vfs/dir/6721f9783ec22414952f74adefb778a9734024c4bdddcdc0e937554dd7cdfc52
1.8G    /var/lib/docker/vfs/dir/b6a42cc371e1b434dc0e5be1ed055415733ad357aa5b55d73761290c3e79cccd
2.1G    /var/lib/docker/vfs/dir/d3ce012e587e0e30e71bbcef51529e924b483bcad588ac77242916429b9816b3
1.8G    /var/lib/docker/vfs/dir/d3ce012e587e0e30e71bbcef51529e924b483bcad588ac77242916429b9816b3-init
134M    /var/lib/docker/vfs/dir/fa0feaa5e6a458b2aa4564cd4c72f21b9ef2c822b65aeadf008a0b21b82de7f8
1.8G    /var/lib/docker/vfs/dir/fa170a62ba9e7d0ce63e476b0f120266448158b5e7b088bf193b79eda7f08d01

That is over 12GB of unaccounted for space being used.

Output from ncdu:

Describe the results you expected:
Docker should list all layers still in storage in the vfs/ directory.

Additional information you deem important (e.g. issue happens only occasionally):
Note that this seems to only happen when using docker in docker.

It should be noted that this build is only about 500mb. There is no reason to be using 12Gb, and I can't seem to find out where all of this is coming from.

Output of docker version:

Docker version 17.12.0-ce, build c97c6d6

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 17.12.0-ce
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-5-amd64
Operating System: Ubuntu 16.04.3 LTS (containerized)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 492.5MiB
Name: 92f9d719f428
ID: TSGE:4SAZ:DERM:ZDYI:4KFJ:V44F:RBKJ:RXL7:72KB:FEXW:AY2H:YCWI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
Vultr VPS with Docker in Docker.

[braunsonm]

To add further info, I don't think it's an issue with my build. Running the same build on windows results in the image size being normal. And showing up in docker system df -v

(not sure why it created a with 2.5Gbs, but regardless the final image size is only 384mb)

[braunsonm]

This issue also affects docker system prune -a --volumes

WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all volumes not used by at least one container
        - all images without at least one container associated to them
        - all build cache
Are you sure you want to continue? [y/N] y
Deleted Containers:
e81177326b98d63a0f920abe92d0cc0cd699f860ed82f8367ccfe14b6e4ea45a

Deleted Images:
untagged: microsoft/dotnet:2-sdk
untagged: microsoft/dotnet@sha256:c5916d40aa6b210fa7e587bf4cb890615177e70ab4196d702e29addd3a29ffdc
deleted: sha256:dbcf26a1afe36cca5c3e61ba8b634efd032be283c833ad1a65d32ff36a38e72d
deleted: sha256:0582648f27bc67dc228f5791f01709dc749bad3b550375e11f7fedb5a7c682b1
deleted: sha256:78de533061349320fe13152b57b1b15f22285b60396c1f7c29a6e72331a38128
deleted: sha256:9969575612df19d22606980b2ef55e813488ba4e3a95e3e1581e3f9cf149882e

Total reclaimed space: 354.7MB

root@92f9d719f428:/# du -sh /var/lib/docker/vfs/dir/*
1.8G    /var/lib/docker/vfs/dir/1f6bee7f0e35932bed711496b14cb58fbb07017c5a98fcf5893bb924e89c2ea4
288M    /var/lib/docker/vfs/dir/2bea78b30abf2e5137e91d963c1980d90c4a9dd58bc4e019710f646ad13548eb
111M    /var/lib/docker/vfs/dir/416dedbc38925c3370753f742e8b8c326a04d1cbaa5302c42eb8f8f28d49b9aa
628M    /var/lib/docker/vfs/dir/5d437a35347ec637156103ed49639481a3f4608adcf59df66685d4f032d83e7b
319M    /var/lib/docker/vfs/dir/6272847cccecd9865fbf7f401caafdf3db65127c56439760d5a2674fd0c6a7da
141M    /var/lib/docker/vfs/dir/6721f9783ec22414952f74adefb778a9734024c4bdddcdc0e937554dd7cdfc52
1.8G    /var/lib/docker/vfs/dir/b6a42cc371e1b434dc0e5be1ed055415733ad357aa5b55d73761290c3e79cccd
134M    /var/lib/docker/vfs/dir/fa0feaa5e6a458b2aa4564cd4c72f21b9ef2c822b65aeadf008a0b21b82de7f8
1.8G    /var/lib/docker/vfs/dir/fa170a62ba9e7d0ce63e476b0f120266448158b5e7b088bf193b79eda7f08d01

They remained after a docker prune.

[jethrogb]

I'm seeing something similar with

Docker version 17.03.1-ce, build c6d412e

Here's what Docker reports:

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              19                  19                  2.93 GB             21.17 MB (0%)
Containers          41                  41                  5.332 GB            0 B (0%)
Local Volumes       1                   1                   4.933 GB            0 B (0%)

Here's the actual disk usage:

# du -d1 -mx /var/lib/docker
1    ./swarm
1    ./network
13    ./image
1    ./tmp
8046    ./aufs
4664    ./volumes
15179    ./containers
1    ./trust
1    ./plugins
27900    .

Note the 14GB discrepancy

[cpuguy83]

The "vfs" dir has nothing to do with build cache.
It sounds like you are using the vfs storage driver which consumes a lot of space since it copies everything (no layering supported).
There may be extra data in there from failed removes (which in the past were silently ignored).

@jethrogb
Indeed docker system df only takes into account container fs size and not ancillary things such as log data.

[jethrogb]

@cpuguy83 yup, logs were it in my case, thanks! I think docker should show this. I guess this is moby/moby#32205

[cpuguy83]

@jethrogb Agreed and yep!

For the OP's issue of having extra layer dirs lying around, on moby master there is a garbage collector which runs on daemon start which removes these unreferenced layers.
This is not in the most recent release (19.03), though there is some discussion on backporting it here: docker/engine#268

Going to close this one as a non-issue.

[alexanderadam]

Indeed docker system df only takes into account container fs size and not ancillary things such as log data.

What else is relevant and not taken into account?

This also seems to be an issue that pops up every now and then. Issue #38848 seems to be a duplicate.