Skip to content
This repository was archived by the owner on Aug 18, 2025. It is now read-only.

Commit 1dc4c3d

Browse files
Jonathan YuKatie Horne
Jonathan Yu
and
Katie Horne
authored
chore: document inotify watch limit (#183)
Add a guide explaining the inotify watch limit problem, including instructions to diagnose and resolve it. Co-authored-by: Katie Horne <katie@coder.com>
1 parent fa0ffc5 commit 1dc4c3d

File tree

3 files changed

+287
-0
lines changed

3 files changed

+287
-0
lines changed

‎guides/troubleshooting/index.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
title: "Troubleshooting"
3+
description: Learn how to troubleshoot Coder-related errors.
4+
---
5+
6+
<children></children>

‎guides/troubleshooting/inotify-watch-limits.md‎

Lines changed: 275 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,275 @@
1+
---
2+
title: inotify Watcher Limit Problems
3+
description: Learn how to resolve problems related to the inotify Watcher Limit.
4+
---
5+
6+
When using some applications and tools, including Webpack or [code-server], you
7+
may encounter an error similar to the following:
8+
9+
> Watchpack Error (watcher): Error: ENOSPC: System limit for number of file
10+
> watchers reached, watch '/some/path'
11+
12+
[code-server]: https://github.com/cdr/code-server
13+
14+
This article will show you how to diagnose and troubleshoot this error, which
15+
relates to an elevated number of inotify watchers in use.
16+
17+
## Background
18+
19+
[`inotify`] allows programs to monitor files for changes, so that they receive
20+
an event whenever a user or program modifies a file. `inotify` requires kernel
21+
resources (memory and processor) for each file it tracks. As a result, the Linux
22+
kernel limits the number of file watchers that each user can register. The
23+
default settings vary according to the host system distribution; on Ubuntu 20.04
24+
LTS, the default limit is 8,192 watches per instance.
25+
26+
[`inotify`]: https://en.wikipedia.org/wiki/Inotify
27+
28+
On a 64-bit system, each `inotify` watch that programs register will consume
29+
~1 kB of kernel memory, which cannot be swapped to disk and is not
30+
counted against the environment memory limit setting.
31+
32+
## Diagnosis
33+
34+
If you encounter the error that's the focus of this article, the total number of
35+
watchers in use is approaching the `max_user_watches` setting. The following
36+
sections will show you how to verify if this is the case.
37+
38+
### Check tunable settings
39+
40+
There are three kernel tuning options related to the `inotify` system:
41+
42+
- `fs.inotify.max_queued_events`: The upper bound on the number of file
43+
notification events pending delivery to programs
44+
- `fs.inotify.max_user_instances`: The maximum number of `inotify` instances per
45+
user (programs using `inotify` will typically create a single _instance_, so
46+
this limit is unlikely to cause issues)
47+
- `fs.inotify.max_user_watches`: The maximum number of files and folders that
48+
programs can monitor for changes
49+
50+
To see the values for these settings that are applicable to your environment,
51+
run:
52+
53+
```console
54+
$ sysctl fs.inotify.{max_queued_events,max_user_instances,max_user_watches}
55+
fs.inotify.max_queued_events = 16384
56+
fs.inotify.max_user_instances = 128
57+
fs.inotify.max_user_watches = 8192
58+
```
59+
60+
Because these settings are not namespace-aware, the values will be the same
61+
regardless of whether you run the commands on the host system or inside a
62+
container running on that host.
63+
64+
> See [inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html) for
65+
> additional details regarding the `inotify` system.
66+
67+
### Identify inotify consumers
68+
69+
To identify the programs consuming `inotify` watches, you can use a script that
70+
summarizes the information available in the `/proc` filesystem, such as
71+
[`inotify-consumers`].
72+
73+
This script will show the names of programs along with the number of `inotify`
74+
watches registered with the kernel:
75+
76+
```console
77+
$ ./inotify-consumers
78+
INOTIFY
79+
WATCHER
80+
COUNT PID USER COMMAND
81+
--------------------------------------
82+
269 254560 coder /opt/coder/code-server/lib/node /opt/coder/code-server/lib/vscode/out/bootstrap-fork --type=watcherService
83+
5 1722 coder /opt/coder/code-server/lib/node /opt/coder/code-server/lib/vscode/out/vs/server/fork
84+
2 254538 coder /opt/coder/code-server/lib/node /opt/coder/code-server/lib/vscode/out/bootstrap-fork --type=extensionHost
85+
2 1507 coder gpg-agent --homedir /home/coder/.gnupg --use-standard-socket --daemon
86+
87+
278 WATCHERS TOTAL COUNT
88+
```
89+
90+
> Please note that this is a third-party script published by an individual who
91+
is not affiliated with Coder, and as such, we cannot provide a warranty or
92+
support for its usage.
93+
94+
[`inotify-consumers`]:
95+
https://github.com/fatso83/dotfiles/blob/master/utils/scripts/inotify-consumers
96+
97+
To see the specific files that the tools track for changes, you can use `strace`
98+
to monitor invocations of the `inotify_add_watch` system call:
99+
100+
```console
101+
$ strace --follow-forks --trace='inotify_add_watch' inotifywait --quiet test
102+
inotify_add_watch(3, "test", IN_ACCESS|IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_CLOSE_NOWRITE|IN_OPEN|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF) = 1
103+
```
104+
105+
This example shows that the `inotifywait` command is listening for notifications
106+
related to the `test` file.
107+
108+
## Resolution
109+
110+
If you encounter the file watcher limit, you can do one of two things:
111+
112+
1. Reduce the number of file watcher registrations
113+
1. Increase the maximum file watcher limit
114+
115+
We recommend attempting to reduce the file watcher registrations first, because
116+
increasing the number of file watches may result in high processor utilization.
117+
118+
### Reduce watchers
119+
120+
Many applications include files that change rarely (e.g., third-party
121+
dependencies stored in `node_modules`). Your tools may watch for changes to
122+
these files and folders, consuming `inotify` watchers. These tools typically
123+
provide configuration settings to exclude certain files, paths, and patterns
124+
from file watching.
125+
126+
For example, Visual Studio Code and `code-server` apply the following [user
127+
workspace setting] by default:
128+
129+
```json
130+
"files.watcherExclude": {
131+
"**/.git/objects/**": true,
132+
"**/.git/subtree-cache/**": true,
133+
"**/node_modules/**": true,
134+
"**/.hg/store/**": true
135+
},
136+
```
137+
138+
Consider adding other infrequently-changed files to this list, which will cause
139+
Visual Studio Code to poll (or check periodically) for changes to those files.
140+
141+
[user workspace setting]: https://code.visualstudio.com/docs/getstarted/settings
142+
143+
For information on how to do this with other software tools, please see their
144+
documentation/user manuals.
145+
146+
### Increase the watch limit
147+
148+
You can increase the kernel tunable option to increase the maximum number of
149+
`inotify` watches for each user. This is a global setting that applies to all
150+
users sharing the same system/Kubernetes node. To do this, modify the `sysctl`
151+
configuration file, or apply a DaemonSet to the Kubernetes cluster to apply that
152+
change to all nodes automatically.
153+
154+
For example, you can create a file called `/etc/sysctl.d/watches.conf` and
155+
include the following contents:
156+
157+
```text
158+
fs.inotify.max_user_watches = 65536
159+
```
160+
161+
Alternatively, you can use the following DaemonSet with `kubectl apply`:
162+
163+
```yaml
164+
apiVersion: apps/v1
165+
kind: DaemonSet
166+
metadata:
167+
name: more-fs-watchers
168+
namespace: kube-system
169+
labels:
170+
app: more-fs-watchers
171+
k8s-app: more-fs-watchers
172+
spec:
173+
selector:
174+
matchLabels:
175+
k8s-app: more-fs-watchers
176+
template:
177+
metadata:
178+
labels:
179+
name: more-fs-watchers
180+
k8s-app: more-fs-watchers
181+
annotations:
182+
seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
183+
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
184+
spec:
185+
nodeSelector:
186+
kubernetes.io/os: linux
187+
initContainers:
188+
- name: sysctl
189+
image: alpine:3
190+
command:
191+
- sysctl
192+
- -w
193+
- fs.inotify.max_user_watches=16384
194+
resources:
195+
requests:
196+
cpu: 10m
197+
memory: 1Mi
198+
limits:
199+
cpu: 100m
200+
memory: 5Mi
201+
securityContext:
202+
# We need to run as root in a privileged container to modify
203+
# /proc/sys on the host (for sysctl)
204+
runAsUser: 0
205+
privileged: true
206+
readOnlyRootFilesystem: true
207+
capabilities:
208+
drop:
209+
- ALL
210+
containers:
211+
- name: pause
212+
image: k8s.gcr.io/pause:3.5
213+
command:
214+
- /pause
215+
resources:
216+
requests:
217+
cpu: 10m
218+
memory: 1Mi
219+
limits:
220+
cpu: 100m
221+
memory: 5Mi
222+
securityContext:
223+
runAsNonRoot: true
224+
runAsUser: 65535
225+
allowPrivilegeEscalation: false
226+
privileged: false
227+
readOnlyRootFilesystem: true
228+
capabilities:
229+
drop:
230+
- ALL
231+
terminationGracePeriodSeconds: 5
232+
```
233+
234+
This DaemonSet will ensure that the corresponding pod runs on _every_ Linux node
235+
in the cluster. When new nodes join the cluster, such as during an autoscaling
236+
event, the DaemonSet will ensure that the pod runs on the new node as well.
237+
238+
You can delete the DaemonSet by running:
239+
240+
```console
241+
$ kubectl delete --namespace=kube-system daemonset more-fs-watchers
242+
daemonset.apps "more-fs-watchers" deleted
243+
```
244+
245+
However, note that the setting will persist until the node restarts or another
246+
program sets the `fs.inotify.max_user_watches` setting.
247+
248+
## See also
249+
250+
- [INotify watch
251+
limit](https://blog.passcod.name/2017/jun/25/inotify-watch-limit.html)
252+
provides additional context on this problem and its resolution
253+
- [inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html), the Linux
254+
manual page related to the `inotify` system call
255+
- [Kernel Korner - Intro to inotify](https://www.linuxjournal.com/article/8478)
256+
- [Filesystem notification, part 1: An overview of dnotify and
257+
inotify](https://lwn.net/Articles/604686/) and [Filesystem notification, part
258+
2: A deeper investigation of inotify](https://lwn.net/Articles/605128/)
259+
examine the `inotify` mechanism and its predecessor, `dnotify`, in detail
260+
- Microsoft's Language Server Protocol (LSP) specification [describes an
261+
approach for using file watch
262+
notifications](https://microsoft.github.io/language-server-protocol/specification#workspace_didChangeWatchedFiles)
263+
(Visual Studio Code and code-server, along with many other editors, uses this
264+
protocol for programming language support, and the same constraints and
265+
limitations apply to those tools)
266+
- Resources for Visual Studio Code and code-server:
267+
- [User and Workspace
268+
Settings](https://code.visualstudio.com/docs/getstarted/settings), in
269+
particular, the setting called `files.watcherExclude`
270+
- [VS Code Setting: files.watcherExclude](https://youtu.be/WMNua0ob6Aw)
271+
(YouTube)
272+
- [My ultimate VSCode
273+
configuration](https://dev.to/vaidhyanathan93/ulitmate-vscode-configuration-4i2o),
274+
a blog post describing a user's preferred settings, including file
275+
exclusions

‎manifest.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,12 @@
151151
},
152152
{ "path": "./guides/api.md",
153153
"navigable": false
154+
},
155+
{ "path": "./guides/troubleshooting/index.md",
156+
"navigable": false,
157+
"children": [
158+
{ "path": "./guides/troubleshooting/inotify-watch-limits.md" }
159+
]
154160
}
155161
]
156162
},

0 commit comments

Comments
 (0)