-
Notifications
You must be signed in to change notification settings - Fork 80
chore: add doc on increasing inotify-watch-limits #179
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,96 @@ | ||||||||||
| --- | ||||||||||
| title: Inotify Watch Limit Increase | ||||||||||
| description: Learn how to increase the inotify watcher limit. | ||||||||||
| --- | ||||||||||
|
|
||||||||||
| When running a webpack project, you may encounter an error similar to the | ||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is not Webpack-specific, so I would suggest making this more generic.
Suggested change
|
||||||||||
| following: | ||||||||||
|
|
||||||||||
| ```text | ||||||||||
| Watchpack Error (watcher): Error: ENOSPC: System limit for number of file | ||||||||||
| watchers reached, watch '/some/path' | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| This results from a low number of [inotify | ||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Suggest adding more info about what the inotify mechanism is for, what it does, and the impact of changing the watch limit.
Suggested change
|
||||||||||
| watches](https://confluence.jetbrains.com/display/IDEADEV/Inotify+Watches+Limit) | ||||||||||
| combined with high node usage, causing the log stream to fail. | ||||||||||
|
|
||||||||||
| ## Resolution | ||||||||||
|
|
||||||||||
| Increase the number of file watchers. Because the setting quantifying the number | ||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
| of file watchers isn't namespaced, you'll need to raise the maximum number at | ||||||||||
| the node level. | ||||||||||
|
|
||||||||||
| One way to do this is to use a daemonset in the cluster with a privileged | ||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
DaemonSet is a proper noun so should be capitalized, and we should stylize it as indicated in the official Kubernetes documentation |
||||||||||
| container to set the maximum number of file watchers: | ||||||||||
|
|
||||||||||
| ```yaml | ||||||||||
| apiVersion: apps/v1 | ||||||||||
| kind: DaemonSet | ||||||||||
| metadata: | ||||||||||
| name: more-fs-watchers | ||||||||||
| namespace: kube-system | ||||||||||
| labels: | ||||||||||
| app: more-fs-watchers | ||||||||||
| k8s-app: more-fs-watchers | ||||||||||
| spec: | ||||||||||
| selector: | ||||||||||
| matchLabels: | ||||||||||
| k8s-app: more-fs-watchers | ||||||||||
| template: | ||||||||||
| metadata: | ||||||||||
| labels: | ||||||||||
| name: more-fs-watchers | ||||||||||
| k8s-app: more-fs-watchers | ||||||||||
| spec: | ||||||||||
| hostNetwork: true | ||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @scsmithr Do these really need to share host namespaces? I don't believe so, since sysfs is being mounted read-write. We shouldn't recommend things that don't meet Kubernetes' security recommendations in our docs Especially since this is privileged and running as a DaemonSet, we should try to restrict things as much as possible: https://kubernetes.io/docs/concepts/security/pod-security-standards/ |
||||||||||
| hostPID: true | ||||||||||
| hostIPC: true | ||||||||||
| initContainers: | ||||||||||
| - command: | ||||||||||
| - sh | ||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @scsmithr How come this uses There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I tested this and it works - there's only so much we can do since we need a privileged container to access /proc/sys, but I at least tried to cut down on the host namespaces we were sharing into the DaemonSet pods: apiVersion: apps/v1
kind: DaemonSet
metadata:
name: more-fs-watchers
namespace: kube-system
labels:
app: more-fs-watchers
k8s-app: more-fs-watchers
spec:
selector:
matchLabels:
k8s-app: more-fs-watchers
template:
metadata:
labels:
name: more-fs-watchers
k8s-app: more-fs-watchers
annotations:
seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
nodeSelector:
kubernetes.io/os: linux
initContainers:
- name: sysctl
image: alpine:3
command:
- sysctl
- -w
- fs.inotify.max_user_watches=16384
resources:
requests:
cpu: 10m
memory: 1Mi
limits:
cpu: 100m
memory: 5Mi
securityContext:
# We need to run as root in a privileged container to modify
# /proc/sys on the host (for sysctl)
runAsUser: 0
privileged: true
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
containers:
- name: pause
image: k8s.gcr.io/pause:3.5
command:
- /pause
resources:
requests:
cpu: 10m
memory: 1Mi
limits:
cpu: 100m
memory: 5Mi
securityContext:
runAsNonRoot: true
runAsUser: 65535
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
terminationGracePeriodSeconds: 5 |
||||||||||
| - -c | ||||||||||
| - sysctl -w fs.inotify.max_user_watches=524288; | ||||||||||
| image: alpine:3.6 | ||||||||||
| imagePullPolicy: IfNotPresent | ||||||||||
| name: sysctl | ||||||||||
| resources: {} | ||||||||||
| securityContext: | ||||||||||
| privileged: true | ||||||||||
| volumeMounts: | ||||||||||
| - name: sys | ||||||||||
| mountPath: /sys | ||||||||||
| containers: | ||||||||||
| - resources: | ||||||||||
| requests: | ||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @scsmithr we should set requests/limits too. The pod should be set with a read-only rootfs for security. And I suggest we use the sleep command instead of tail -f? I am actually not sure if we need privileges other than the ability to write to sysfs? We should check this... |
||||||||||
| cpu: 0.01 | ||||||||||
| image: alpine:3.6 | ||||||||||
| name: sleepforever | ||||||||||
| command: ["tail"] | ||||||||||
| args: ["-f", "/dev/null"] | ||||||||||
| volumes: | ||||||||||
| - name: sys | ||||||||||
| hostPath: | ||||||||||
| path: /sys | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| ## Notes | ||||||||||
|
|
||||||||||
| - Daemonsets without node selectors will persist on the cluster and will run on | ||||||||||
| *every* node. Every new node that you spin up will also have an associated | ||||||||||
| daemonset pod spun up to ensure that the sysctl command runs on every node. | ||||||||||
| - You can **delete** this by running: | ||||||||||
|
|
||||||||||
| ```console | ||||||||||
| kubectl delete more-fs-watchers | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| This command removes all related daemonset pods, and no further pods will be | ||||||||||
| spun up. | ||||||||||
|
|
||||||||||
| ## Helpful Resources | ||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
We can't guarantee these will be helpful 😉 |
||||||||||
|
|
||||||||||
| - [Setting Sysctls for a | ||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think this actually helps in this case, because these sysctls are not namespaced and therefore not on the allow list |
||||||||||
| Pod](https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#setting-sysctls-for-a-pod) | ||||||||||
| - [Increasing the Amount of `inotify` | ||||||||||
| Watchers](https://github.com/guard/listen/blob/master/README.md#increasing-the-amount-of-inotify-watchers) | ||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggest titling this "Increasing the inotify Watch Limit" or "Increasing the File Watcher Limit" (I think inotify_add_watch is typically styled in lowercase as it's the name of a system call)
More generally, though, it seems we may want to have this be part of a larger page about tuning options (or more specifically, kernel tuning options) for Coder?