Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nodes constantly running out of inodes #32526

Closed
felipejfc opened this issue Sep 12, 2016 · 14 comments

Comments

Projects
None yet
9 participants
@felipejfc
Copy link

commented Sep 12, 2016

Kubernetes version (use kubectl version):
1.3.5

Environment:

  • Cloud provider or hardware configuration: AWS
  • OS (e.g. from /etc/os-release): Debian
  • Kernel (e.g. uname -a): 4.4.19
  • Install tools: Kops

What happened:
I'm constantly running out of inodes on my k8s nodes, I've used kops to deploy my cluster. Maybe if instead of using ext4 for the root fs we use xfs would fix this problem? @justinsb

Regards

What you expected to happen:
Stop running out of inodes so constantly.

How to reproduce it (as minimally and precisely as possible):
Just use the cluster and make a lot of deploys :)

@dchen1107

This comment has been minimized.

Copy link
Member

commented Sep 12, 2016

@felipejfc Thanks for reporting the issue. In 1.4 release, we detects the out-of-inode issue, and start the eviction to mitigate the issue. In 1.5 release, we are going to add per container inode accounting, so that out-of-inode eviction will handle the situation more gracefully and accurately.

Meanwhile, can you share us the output of docker info? I am wondering which graphdriver you are using here.

@dchen1107

This comment has been minimized.

Copy link
Member

commented Sep 12, 2016

cc/ @justinsb

@dchen1107 dchen1107 added area/kubelet and removed area/kubectl labels Sep 12, 2016

@dchen1107

This comment has been minimized.

Copy link
Member

commented Sep 12, 2016

cc/ @dashpole We are talking about your start project. Here is a real production case.

@felipejfc

This comment has been minimized.

Copy link
Author

commented Sep 12, 2016

here it is @dchen1107

Containers: 45
Running: 45
Paused: 0
Stopped: 0
Images: 26
Server Version: 1.11.2
Storage Driver: overlay
Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.4.19
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.67 GiB
Name: ip-172-21-100-24
ID: C4S6:A3VT:VSRC:5DUB:VXQY:VRRM:TPG5:MV22:F3ZJ:XSZG:B3AV:G6Q3
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: No kernel memory limit support

number of images is low because I've just deleted manually all images that I was not using

@therc

This comment has been minimized.

Copy link
Contributor

commented Sep 24, 2016

There's another issue if you have a lot of large Docker images with more than two layers. That might be the case for environments using CD/CI that rebuild many times a day. Unless you adopt Docker 1.12's new overlay2 storage driver, you'll be using a ton of inodes. In our case, a machine was not seeing any image GC, because it ran out of inodes while still at 75% full, below the 90% GC threshold. In a similar scenario, eviction is not the real solution. A more aggressive GC of old images would work better.

k8s-github-robot pushed a commit that referenced this issue Nov 1, 2016

Kubernetes Submit Queue
Merge pull request #35137 from dashpole/per_container_inode_eviction
Automatic merge from submit-queue

Eviction manager evicts based on inode consumption

Fixes: #32526 Integrate Cadvisor per-container inode stats into the summary api.  Make the eviction manager act based on inode consumption to evict pods using the most inodes.

This PR is pending on a cadvisor godeps update which will be included in PR #35136
@JeanMertz

This comment has been minimized.

Copy link

commented Jan 19, 2017

This issue was closed, because PR #35137 was merged.

However, on our GKE cluster (1.5.1), we're still seeing this happen on a daily basis, and this is on preemptible machines, so they are "refreshed" every 24 hours.

All of the above mentioned by @therc applies to our situation as well; using large images (mainly Ruby, based on Debian, lots of layers), updating many deployments multiple times per day, disk not being an issue but running out of inodes constantly.

Right now, we're seeing one of the kube-dns containers (dnsmasq) failing to start with the error:

dnsmasq: failed to create inotify: No file descriptors available

It has been giving these errors for a couple of hours, but the node is not doing any cleanup.

Looking at the node with df -i, I don't see anything actually wrong (max 17% inode usage), but the pod just won't come online with the same inode error being repeated.

Given my df -i output, this might be totally unrelated to this issue, if so, I can create a new issue.

@JeanMertz

This comment has been minimized.

Copy link

commented Jan 19, 2017

I created a gist with all the relevant output that I can find related to this issue:

https://gist.github.com/JeanMertz/c5f26e69f14910e828f85185987692b2

@JeanMertz

This comment has been minimized.

Copy link

commented Jan 21, 2017

Today, I again noticed inode problems with two of the pods. This time, it's two of the fluentd-cloud-logging-gke pods, see this gist for all the output I could gather:

https://gist.github.com/JeanMertz/cce2c5afc37b8cabefb06da2a0b55b2e

cc: @bowei @airstand since you both asked about more details on the inode problems in #40063 (comment)

@dashpole

This comment has been minimized.

Copy link
Contributor

commented Jan 23, 2017

unexpected error error_class=Systemd::JournalError error=#<Systemd::JournalError: Too many open files>
dnsmasq: failed to create inotify: No file descriptors available

Looks like you have too many files open, and are out of file descriptors. These are different from Inodes. Increasing the file descriptor limit should fix the problem.

@JeanMertz

This comment has been minimized.

Copy link

commented Jan 23, 2017

@dashpole thanks for that pointer.

I'll do some further investigation into this.

However, we are running on GKE, and are using preemptible machines, so any change we make to nodes only stick around for 24 hours at most.

@dashpole

This comment has been minimized.

Copy link
Contributor

commented Jan 23, 2017

You may be able to use startup scripts to make changes to nodes when they start up.

@soudy

This comment has been minimized.

Copy link

commented Jan 26, 2017

It looks like the default file descriptor limit for VMs has been increased in GKE recently.

Fd limit on regular nodes (27 days old)

$ cat /proc/sys/fs/file-max
6175210

And peemptible node (11 hours old):

$ cat /proc/sys/fs/file-max
12358322
@skevy

This comment has been minimized.

Copy link

commented Feb 15, 2017

@JeanMertz did you have any luck with this issue?

@jamshid

This comment has been minimized.

Copy link

commented Mar 7, 2017

Found this on a search, I was having a similar dnsmasq startup failure in a container when starting a large number of them (single docker server, not kubernetes). I think the "Too many open files" message is misleading...

# systemctl status dnsmasq
● dnsmasq.service - Start dnsmasq
...
Mar 07 16:47:33 46ace65ed7e6 dnsmasq[110]: dnsmasq: failed to create inotify: Too many open files
Mar 07 16:47:33 46ace65ed7e6 dnsmasq[110]: failed to create inotify: Too many open files
Mar 07 16:47:33 46ace65ed7e6 dnsmasq[110]: FAILED to start up

If I'm right, the limit being hit is not file handles ("open files" or ulimit -n), the limit is specific to inotify: max_user_instances or max_user_watches. Check your current limits, I was surprised how low it is on Ubuntu 16:

# cat /proc/sys/fs/inotify/max_user_instances
128
# cat /proc/sys/fs/inotify/max_user_watches 
8192

Try increasing them by creating this /etc/sysctl.d/60-max-user-watches.conf file and rebooting:

fs.inotify.max_user_watches = 8192
fs.inotify.max_user_instances = 8192
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.