Increase user watches on nodes

[mbrancato]

What happened:
On a well-sized cluster, I started getting "no space left on device" issues when trying to run kubectl logs on pods.

What you expected to happen:
Logs print successfully, and tail successfully.

How to reproduce it (as minimally and precisely as possible):
Run kubectl logs

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.9.9
• Size of cluster (how many worker nodes are in the cluster?): 4
• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.): Java app, CronJobs
• Others: Kured is deployed and the cluster regularly drains and reboots nodes. Actual filesystem quotas were not full.

It looks like the fix here for me was to increase the fs.inotify.max_user_watches value. The AKS nodes are deployed with the default 8192. Can this value be increased to avoid this issue in the future?

[kvolkovich-sc]

Have the same issue. (k8s 1.11.5)

[paulgmiller]

We saw this too but concluded that inotify objects were being leaked by os/docker and raising the limit only delayed the period between pain. We did a periodic rolling reboot instead (which can be az aks upgrade --same version on aks).
There is some thought that moving to ubuntu 18 will be a better fix (may solve the leak) but aks teams has a decent amount of work to let you configure choosing 16 or 18.
Azure/acs-engine#4251

[mbrancato]

@paulgmiller Kured effectively provides periodic reboots. The systems I noticed this on had an 18 day uptime.

[rubroboletus]

Same problem here, temporary solved by daemonset, which sets fs.inotify.max_user_watches to bigger value using sysctl and also mounts /etc/sysctl.d directory and creates file with fs.inotify.max_user_watches=OUR_VALUE as content.

[vaclav-dvorak]

@rubroboletus we concluded with same "hack" on our clusters. it would be nice not to have to resort to this

[paulgmiller]

Azure/aks-engine#223

[xinyanmsft]

we are running into this limit sometimes, if more of the containers are started with nodemon (these containers are used by engineers to test / debug code)

[xinyanmsft]

more-watches.yaml.txt
I've been using this yaml file to increase the user watches on AKS node. 'kubectl -f more-watches.yaml' and restart the nodes. Hope this helps.

[andrewschmidt-a]

We are experiencing this issue on our clusters. also when we run the above daemon set we get outages in our workloads.

How exactly does it do a rolling update do you have to ensure that there are replica sets that are spreading pods across nodes?

[andrewschmidt-a]

If anyone is interested this is how we are solving it for now. We would love to see a resolution on this issue

name="name_of_aks"
rg="name_of_resource_group"
version=$(az aks show -n $name -g $rg | jq -r ".kubernetesVersion" )

az aks upgrade -k $version -n $name -g $rg -y --no-wait

[palma21]

Increased to fs.inotify.max_user_watches = 1048576

[hameno]

@palma21 Could you please expand on what changed? Has this been rolled out to Azure AKS? If so, which version? How do we get the fix?

[palma21]

Sure
Done in this PR: Azure/aks-engine#1801
Merged in AKS Engine Release: https://github.com/Azure/aks-engine/releases/tag/v0.42.0
Released in AKS in https://github.com/Azure/AKS/releases/tag/2019-11-18

[palma21]

Any upgrade done after the AKS release date (2019-11-18) will have this change (since it's a node setting change it needs an upgrade)

[hameno]

Thanks, will trigger upgrades as soon as possible

[StepanKuksenko]

more-watches.yaml.txt
I've been using this yaml file to increase the user watches on AKS node. 'kubectl -f more-watches.yaml' and restart the nodes. Hope this helps.

@xinyanmsft why do you mount /sys directory to the container ? i tried to configure without this, seems everything is ok. But maybe i don't know something...