Running out of inodes on /run

[byrnedo]

After updating to docker 1.11 I seem to run out of inodes on /run.

Inspecting the /run/docker/libcontainerd dirs I see that my containers have the following numbers of of files:

/var/run/docker/libcontainerd/30b451cdf9a4b92c35e4b042715f0c560b535b0c5af4bc67acf7580baa1b2264
9334
/var/run/docker/libcontainerd/35fd9cdb1f80db3bb5fd16aadfe1b30ead0582906f992e4d4865a81798a15072
256507
/var/run/docker/libcontainerd/3a34cbef697c5b7a3865b4556d97c21427b599ff56ab6833bd02013f71aa7738
35129
/var/run/docker/libcontainerd/5e7e460fcf536f2cd75032a9a92e4843812fa4b72b8b5e13b089a90f1ef28e2a
9942
/var/run/docker/libcontainerd/5ebcfb17c26815533f4abd883fb59344554077018f04ce062a67bbdfe50f5c3d
9942
/var/run/docker/libcontainerd/776106ffe1802805469ff0078e02b71e7c9dd9e8c08e8d31951712c6b8438b49
1002
/var/run/docker/libcontainerd/91f43f1dd0252f30363337a4c073962194b9cc7fb1df632a92db4f2fa15ca551
55663
/var/run/docker/libcontainerd/b417ff5d733d5f2bc37cac57bb7af72b86bc680f44e7d5656c55d1067ce66c5d
46336
/var/run/docker/libcontainerd/bd07d50b3c3bdf7e9c8ddf6894e5a079af09f7c9d0d9e27f7d62065cdf181618
9942
/var/run/docker/libcontainerd/ce6bf9750bd4183f2b0d87cf67d11c4047e0473cf396867b70798cccd818e67f
10705
/var/run/docker/libcontainerd/d59d15e94c33556ae099232caed4d824c966e754a2ae63116f3a4249ef290507
7581
/var/run/docker/libcontainerd/docker-containerd.pid
1
/var/run/docker/libcontainerd/docker-containerd.sock
1
/var/run/docker/libcontainerd/e6f8f0e6de324d13389f167c68a31c345eb740b0b8b60b3995a170940e38d01f
26
/var/run/docker/libcontainerd/eb7baaf652b9247f9efbadebbdce9b6b3d88b1515ab71462b91b995187496373
35275
/var/run/docker/libcontainerd/event.ts
1

The container with the most files is one I use to 'exec' commands on for health checking within an overlay network.
Restarting the container resolves the issue
Listing the files within that culprit container's dir in /var/run/docker/libcontainerd I see piles of

...
123e29754fba075637b49a21362c5265e5c350b5aa516c187cdc498f8d365a01-stdin
123e29754fba075637b49a21362c5265e5c350b5aa516c187cdc498f8d365a01-stdout
...

I also updated kernel recently as well

Output of uname -a:

Linux aws-qa-node-1 4.2.0-35-generic #40~14.04.1-Ubuntu SMP Fri Mar 18 16:37:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Trying to exec a command on a container:

sudo docker exec dev_consulcheck_blue_1 /bin/sh
mkfifo: /var/run/docker/libcontainerd/35fd9cdb1f80db3bb5fd16aadfe1b30ead0582906f992e4d4865a81798a15072/041ced8a2353434d55e034a37d7c096e4030bbf1878176331c69d1666af3990f-stdin no space left on device

Trying to run a container:

sudo docker run --rm -it busybox /bin/sh
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
385e281300cc: Pull complete 
a3ed95caeb02: Pull complete 
Digest: sha256:4a887a2326ec9e0fa90cce7b4764b0e627b5d6afcb81a3f73c85dc29cea00048
Status: Downloaded newer image for busybox:latest
docker: Error response from daemon: mkdir /var/run/docker/libcontainerd/557517143c7a394f2bdd0f2719274d596682b4eaa180e56c176ae2770e10e815/rootfs: no space left on device

Output of docker version:

Client: 
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:34:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:34:23 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 13
 Running: 13
 Paused: 0
 Stopped: 0
Images: 69
Server Version: 1.11.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 397
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local rexray
 Network: host bridge null overlay
Kernel Version: 4.2.0-35-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.952 GiB
Name: aws-qa-node-1
ID: A5QZ:FCVL:NXOH:5V3R:SKZ4:V3EZ:55D2:EPLE:3QBM:TQXK:EXEL:MFSI
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
 nodeindex=1
 role.service=true
 role.infra=true
 provider=generic
 ec2.instance.type.t2.small=true
Cluster store: consul://10.0.2.95:8500
Cluster advertise: 10.0.4.203:2376

Additional environment details (AWS, VirtualBox, physical, etc.):

Running on aws.

[thaJeztah]

Thanks for reporting!

ping @crosbymichael @anusha-ragunathan any ideas on this one?

[icecrime]

Thank you very much for the detailed report! Cc @tonistiigi too.

[byrnedo]

No problem, I notice that this is happening locally too on the same setup (ubuntu 14.04, same kernel, same docker version), those stdin and stdout files keep gathering with every exec done on a container.

[justincormack]

The three pipes for each exec for stdin, stdout and stderr are not deleted after each docker exec, so the number of files increases each time by 3. Will look and see where these should be deleted.

[justincormack]

Ok, I will fix this.

[icecrime]

Escalating this issue to P0: we're going to move forward with a 1.11.1 patch release for this.

[yeasy]

Is this fixed in new release?
What version should be safe?
Thanks!

[thaJeztah]

@yeasy it's fixed in docker 1.11.1

[yeasy]

@thaJeztah
Thanks, then we will move to 1.11.1.

[yeasy]

I've to reopen this issue, as when using 1.11.2.
After lots of docker exec, the inode number keeps increasing in /var/run, till exhausted.

[antoniomercado]

I'm experiencing the same issue. Here is a list of the top 10 largest directories in /run/docker/libcontainerd

988K    /run/docker/libcontainerd/154d830d812a3b1c01fcc2a57edc61690d36db123506c312b8fd1ca1aaff1f34/rootfs/src/node_modules/grunt-jsdoc/node_modules/jsdoc/templates/default/static/fonts
988K    /run/docker/libcontainerd/154d830d812a3b1c01fcc2a57edc61690d36db123506c312b8fd1ca1aaff1f34/rootfs/src/node_modules/grunt-jsdoc/node_modules/jsdoc/out/fonts
984K    /run/docker/libcontainerd/154d830d812a3b1c01fcc2a57edc61690d36db123506c312b8fd1ca1aaff1f34/rootfs/usr/share/perl/5.18.2/ExtUtils
981M    /run/docker/libcontainerd/154d830d812a3b1c01fcc2a57edc61690d36db123506c312b8fd1ca1aaff1f34/rootfs
981M    /run/docker/libcontainerd/154d830d812a3b1c01fcc2a57edc61690d36db123506c312b8fd1ca1aaff1f34
981M    /run/docker/libcontainerd/
980K    /run/docker/libcontainerd/154d830d812a3b1c01fcc2a57edc61690d36db123506c312b8fd1ca1aaff1f34/rootfs/usr/local/lib/node_modules/grunt-cli/node_modules/findup-sync/node_modules/lodash
980K    /run/docker/libcontainerd/154d830d812a3b1c01fcc2a57edc61690d36db123506c312b8fd1ca1aaff1f34/rootfs/src/node_modules/load-grunt-config/node_modules/lodash
980K    /run/docker/libcontainerd/154d830d812a3b1c01fcc2a57edc61690d36db123506c312b8fd1ca1aaff1f34/rootfs/src/node_modules/joi

Notice how there are node_modules folders listed. This alludes to a relationship of these files to volumes in my container. What is weird though is that I've configured docker to use a different mount point for images/containers that is attached to a separate volume from the root volume of the host machine, so why would these files be affecting the root volume? In my /etc/default/docker file I have this DOCKER_OPTS="-g /mnt/docker"
It sure would be nice if docker utilized the mount point specified by these options rather than the host machine for these libcontainerd files.

Running docker version 1.11.0 here, but since @yeasy points out that the issue still exists in 1.11.2 I didn't bother upgrading.

[thaJeztah]

@antoniomercado what distro and version are you on? Are you sure your /etc/default/docker is used? If you're running on a system that uses systemd, that file is not used

[thaJeztah]

ping @tonistiigi ^^

[antoniomercado]

@thaJeztah I'm using Ubuntu 14.04 which doesn't use systemd by default. I think Ubuntu switched to systemd by default in version 16.04, so I don't think that's the issue. Plus I know its using the mount point by looking at the df -h output.

[tonistiigi]

@antoniomercado Please upgrade to v1.11.2. @yeasy If you can reproduce leftover inodes with v1.11.2 please report how you do it and where the leftover inodes are.

@antoniomercado The files in your log are new bind mounts from /var/lib/docker(/mnt/docker if you set -g) and do not count against the inode limits of /run. Also, this bind mount is removed in v1.11.2.

[antoniomercado]

@tonistiigi I've discussed this internally with engineering at my organization and we are scheduled to upgrade our production servers to 1.11.2 in the near term, I will report back if any issues. Thanks for your feedback.

[averri]

I'm having the same issue on CentOS 7.4, Docker version 17.05.0-ce, build 89658be:

[root@home]# docker run ubuntu echo hello
docker: Error response from daemon: mkdir /var/run/docker/libcontainerd/0823930a8dba111dc94265d898c0b44b1ae4b179a9c6672bc12e8cec2eab6b1d: no space left on device.

[thaJeztah]

@averri be aware that Docker 17.05 reached end-of-life in June of last year, and hasn't been maintained since then.

[averri]

Hi @thaJeztah, thanks for informing. The workaround here is to delete unused files in order to free the inodes.

[rishiloyola]

I am facing the same issue. It is generating the following files.

Docker version - Docker version 17.09.0-ce, build afdb6d4

[rishiloyola]

BUMP 😄

[thaJeztah]

@rishiloyola are you still able to reproduce this on a current version of Docker? Docker 17.09 reached EOL in december 2017. If you're still able to reproduce, could you open a new issue instead? The original issue reported here was fixed in 2016. Since then, the entire runtime has been replaced, so although the problem may look similar, there's a high chance that the cause is different, so better to open an new issue instead.

[das-intensity]

Now @thaJeztah is correct that nobody should be using docker 17 anymore, BUT for those that are, here's how I solved this...

The issue is docker failed to cleanup files such as:

prwx------ 1 root root 0 May 13 22:55 /run/docker/libcontainerd/37babdc63eacecd1e994d670af51fa83715b86042463fa7dbdbca81ad4bcae90/147b2ff0c0ea6ab30a21dd3cffb0e65bde6d98d051ca381e83a7c4d9935c4350-stdout

Create /root/docker-cleanup.sh with:

#!/bin/bash

set -e

# Notes:
# - The *1* avoids hitting init-stdout etc
# - The -mtime +7 avoids hitting recent files
# - The lsof call means that if the file is currently in-use, it won't be deleted
# None of this is "correct" or "safe", but should be sufficient to prevent /run from running out of inodes
find /run/docker/libcontainerd/ -name "*1*-std*" -mtime +7 -exec bash -c "lsof {} || rm -fv {}"  \;

Then add this cron job:

0 0 * * * /root/docker-cleanup.sh >/dev/null

Once a day it'll go through and delete any old unused files.

This was:

• Created within a few minutes
• To solve the problem that I was having
• Will surely blow up someone's production database if they use it

Do the right thing and update your docker version, but otherwise, enjoy at your own risk ;)