containerd-shim creates many inotify instances on AlmaLinux VM

[gautamnimmala]

Environmental Info:
K3s Version:

k3s -v
k3s version v1.30.1+k3s1 (80978b5b)
go version go1.22.2

OS Version:

cat /etc/redhat-release 
AlmaLinux release 9.4 (Seafoam Ocelot)

Node(s) CPU architecture, OS, and Version:

uname -a
Linux hostname 5.14.0-427.18.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 28 06:27:02 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
single, all in one node

Containerd Version:

/var/lib/rancher/k3s/data/current/bin/containerd --version
containerd github.com/k3s-io/containerd v1.7.15-k3s1

Describe the bug:

When running k3s on a AlmaLinux machine with cgroup2 support, the error message failed to create fsnotify watcher: too many open files occurs.

After investigating, it was found that containerd-shim processes are consuming a large number of inotify instances.
Command used to find the containerd-shim processes

for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | cut -d/ -f3 | xargs -I '{}' -- ps --no-headers -o comm {} | sort | uniq -c | sort -nr

Increasing fs.inotify.max_user_instances = 256 temporarily resolved the issue, but the problem persists.

The node's default values for inotify are

fs.inotify.max_user_watches = 478130
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
soft nofile ulimit = 1024

I have read through issue #10020 on GitHub, which suggested that this issue was resolved in k3s version 1.30.1. However, the problem persists in my setup.

Steps To Reproduce:

• Deploy a single node k3s cluster on an AlmaLinux machine with K3s version: v1.30.1
• Check the installed cgroup and make sure the current node supports cgroup2

grep cgroup /proc/filesystems
nodev	cgroup
nodev	cgroup2

• When the number of pods in the cluster is large, you will encounter the error Too many open files.
• Use the below command to monitor the usage of inotify instances:
  for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | cut -d/ -f3 | xargs -I '{}' -- ps --no-headers -o comm {} | sort | uniq -c | sort -nr
• Notice that the containerd-shim processes are consuming a large number of inotify instances compared to other processes.
• Result from my machine

for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | cut -d/ -f3 | xargs -I '{}' -- ps --no-headers -o comm {} | sort | uniq -c | sort -nr
     99 containerd-shim
     14 k3s-server
      8 systemd
      4 nginx-ingress-c
      3 manager
      2 agetty
      2 NetworkManager
      1 systemd-udevd
      1 systemd-logind
      1 fluent-bit
      1 external-secret
      1 dbus-broker-lau
      1 crond
      1 containerd
      1 configmap-reloa
      1 adapter

Expected behavior:

• The k3s version should be v1.30.1
• The containerd version should be v1.7.15-k3s1
  I expect that the number of inotify instance used by containerd-shim processes should be with in reasonable limits and not lead to Too many open files error messages.
  Actual behavior:

The containerd-shim processes are consuming a disproportionately high number of inotify instances, leading to the Too many open files error
Additional context / logs:

• Increasing user.max_inotify_instances temporarily resolved the issue, but the root cause remains that containerd-shim processes are using too many inotify instances.

[brandond]

I have read through issue #10020 on GitHub, which suggested that this issue was resolved in k3s version 1.30.1.

No, that's not what it suggests: #10020 (comment)

Perhaps we just need to document that containerd requires a higher number of inotify watches than some distros allow by default?