Bloated Data loop file (/var/lib/docker/devicemappper/devicemapper/data)

[coeotic]

Description
This IS, perhaps, a duplicate of #3182 or #9786... but I do not see any solution/resolution that applies to my situation.

I need information to help me diagnose a bloated Data loop file. Within 2 days of operation, my instance store is full, but I have no idea why.

Steps to reproduce the issue:

1. I'm not sure. That is why I'm asking for assistance with debugging the bloated data file.

I run two images on my instance, with one mounting a volume from the other, using a --volumes-from argument. Both instances have live binaries, with one writing to the 'shared' volume to have its contents show in the other.

Describe the results you received:
Within 2 days, Data loop file is stuffed with more than 6gb of... who knows what.

Describe the results you expected:
To have less than 6gb of... who knows what filling my Data loop file. I would expect to do this with some help (e.g. troubleshooting guidance), to understand what is being stored there.

Additional information you deem important (e.g. issue happens only occasionally):
No other docker deploy in our system produces 6gb of data. I am looking for a way to understand what is buffering up in the data files.

Output of docker version:

Docker version 1.9.1, build a34a1d5

Output of docker info:

Containers: 2
Images: 52
Server Version: 1.9.1
Storage Driver: devicemapper
Pool Name: docker-202:1-154808-pool
Pool Blocksize: 65.54 kB
Base Device Size: 107.4 GB
Backing Filesystem:
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 6.462 GB
Data Space Total: 107.4 GB
Data Space Available: 16.78 MB
Metadata Space Used: 6.144 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 16.78 MB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-96-generic
Operating System: Ubuntu 14.04.5 LTS
CPUs: 2
Total Memory: 7.337 GiB
Name: puppet-master-docker-i-aa53a01e.stage.us.quixey.com
ID: FMEW:U47R:XA7V:RBVS:DXRP:ALPQ:D67A:QB3X:KDSD:DQML:VE44:6E7B
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

EC2 instance with a standard docker run script for two containers.

[thaJeztah]

who knows what filling my Data loop file. I would expect to do this with some help (e.g. troubleshooting guidance), to understand what is being stored there.

The data loop file stores the filesystems of all containers; if a container is writing a lot of data (think of a container that's writing to log-files instead of to stdout/stderr), space gets occupied by that data.

As an initial way to see what potentially takes up that space, you can do docker diff <container-id> to see which files are modified or created inside the container since it was started. For debugging, you can then docker exec into the container to check the size of those files/directories.

For logging, it's generally recommended to make your container's process write to stdout/stderr, so that you can use Docker's native logging drivers to collect logs (instead of writing them to a file in the container). Not al software does this by default, in which case you could (for example) symlink the log-file, for example like is done in the official nginx image.

Also, if you're using the default JSON-file logging driver, make sure to specify --max-size and/or --max-files to prevent the log files from growing too big https://docs.docker.com/engine/admin/logging/overview/#/json-file-options

If you still want to log to regular files, make sure to use a volume for the directory that the log files are written to.

I don't suspect there's a bug here, and without knowing what's running in the container, and how the container is configured, it's difficult to tell.

[CluEleSsUK]

I've noticed the same issue - all my containers write to stdout, and the docker diff <container-id> shows no changes. I've had to remove old images with nearly every deploy as the disk footprint is so large

[tjliupeng]

I also get this issue. The /var/lib/docker occupies 16G and no more containers can be created. Only three images on this node.

[root@itsma-kuber-node1 ~]# docker images
REPOSITORY                                     TAG                 IMAGE ID            CREATED             SIZE
gcr.io/google_containers/hyperkube             v1.3.2              d64785c49d87        3 months ago        416.3 MB
gcr.io/google_containers/pause-amd64           3.0                 99e59f495ffa        5 months ago        746.9 kB
gcr.io/google_containers/kube-registry-proxy   0.3                 b86ac3f11a0c        14 months ago       151.2 MB

[root@itsma-kuber-node1 ~]# df -h /var
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   44G   44G  774M  99% /

[root@itsma-kuber-node1 ~]# docker info
Containers: 6
 Running: 6
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.797 GiB
Name: itsma-kuber-node1
ID: 3UN3:4ZIX:AOEN:TEKR:XCVE:UD7W:ZH2D:2PNG:BZVB:TJGH:3Q7K:7XAX
Docker Root Dir: /opt/kubernetes/data/docker
Debug mode (client): false
Debug mode (server): false
Http Proxy: ***
No Proxy: localhost,127.0.0.1/8,***
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled

[thaJeztah]

@tjliupeng you're reporting with a different graph driver (overlay), which is not related to this issue. Also note that /var/lib/docker stores more than images and containers (also log files, volume) which can take up space

[thaJeztah]

anyway, I'm closing this issue as we didn't get more information from the original reporter, and it's likely there's no bug here.

Note that for docker 1.13, we're adding data management commands to make cleaning up unused things easier