From b06d7f90abf2fe402d52c4d8f4471030bf4f387d Mon Sep 17 00:00:00 2001 From: Jonathan Yu Date: Mon, 22 Mar 2021 16:26:04 +0000 Subject: [PATCH 1/9] chore: document inotify watch limit Add a guide explaining the inotify watch limit problem, including instructions to diagnose and resolve it. --- guides/admin/inotify-watch-limits.md | 266 +++++++++++++++++++++++++++ manifest.json | 3 +- 2 files changed, 268 insertions(+), 1 deletion(-) create mode 100644 guides/admin/inotify-watch-limits.md diff --git a/guides/admin/inotify-watch-limits.md b/guides/admin/inotify-watch-limits.md new file mode 100644 index 000000000..4c418832b --- /dev/null +++ b/guides/admin/inotify-watch-limits.md @@ -0,0 +1,266 @@ +--- +title: Troubleshooting inotify Watcher Limit Problems +description: Learn how to resolve problems related to the inotify Watcher Limit. +--- + +The [`inotify` facility] allows programs to monitor files and directories for +changes, so that they will receive an event immediately whenever a user or +program modifies the file or directory. Many such programs can fall back to +periodically checking files and directories for changes, also known as polling. +Using polling avoids the watch limit, but may result in higher system loading +and a longer interval for the program to detect changes. + +`inotify` requires kernel resources (memory and processor) for each file it +tracks. As a result, the Linux kernel limits the number of file watchers that +each user can register. The default settings vary according to the host system +distribution; on Ubuntu 20.04 LTS, the default limit is 8,192 watches per instance + +[`inotify` facility]: https://en.wikipedia.org/wiki/Inotify + +With some applications and tools, including Webpack or [code-server], you may +encounter an error similar to the following: + +> Watchpack Error (watcher): Error: ENOSPC: System limit for number of file +> watchers reached, watch '/some/path' + +[code-server]: https://github.com/cdr/code-server + +On a 64-bit system, each `inotify` watch that programs register will consume +approximately 1 kB of kernel memory, which cannot be swapped to disk and is not +counted against the environment memory limit setting. + +## Diagnosis + +If the total number of watchers is close to the `max_user_watches` setting, then +it is likely that you are encountering this limit. + +### Check tunable settings + +There are three kernel tuning options related to the `inotify` system: + +- `fs.inotify.max_queued_events`, which defines an upper bound on the number of + file notification events pending delivery to programs. +- `fs.inotify.max_user_instances`, which determines the maximum number of + `inotify` instances per user. Programs using `inotify` will typically create + a single _instance_, so this limit is unlikely to cause issues. +- `fs.inotify.max_user_watches`, which defines the maximum number of files and + folders that programs can monitor for changes. + +For additional detail regarding the `inotify` system, please see +[inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html). + +To see the values for these settings applicable to your environment, run the +following commands: + +```console +$ sysctl fs.inotify.{max_queued_events,max_user_instances,max_user_watches} +fs.inotify.max_queued_events = 16384 +fs.inotify.max_user_instances = 128 +fs.inotify.max_user_watches = 8192 +``` + +Because these settings are not namespace-aware, the values will be the same +regardless of whether you run the commands on the host system or inside a +container running on that host. + +### Identify inotify consumers + +To identify the programs consuming `inotify` watches, you can use a script that +summarizes the information available in the `/proc` filesystem, such as +[`inotify-consumers`]. This script will show the names of programs along with +the number of `inotify` watches registered with the kernel: + +```console +$ ./inotify-consumers + INOTIFY + WATCHER + COUNT PID USER COMMAND +-------------------------------------- + 269 254560 coder /opt/coder/code-server/lib/node /opt/coder/code-server/lib/vscode/out/bootstrap-fork --type=watcherService + 5 1722 coder /opt/coder/code-server/lib/node /opt/coder/code-server/lib/vscode/out/vs/server/fork + 2 254538 coder /opt/coder/code-server/lib/node /opt/coder/code-server/lib/vscode/out/bootstrap-fork --type=extensionHost + 2 1507 coder gpg-agent --homedir /home/coder/.gnupg --use-standard-socket --daemon + + 278 WATCHERS TOTAL COUNT +``` + +Please note that this is a third-party script published by an individual who is +not affiliated with Coder, and as such, we cannot provide a warranty or support +for its usage. + +[`inotify-consumers`]: https://github.com/fatso83/dotfiles/blob/master/utils/scripts/inotify-consumers + +To see the specific files and directories that the tools track for changes, you +can use `strace` to monitor invocations of the `inotify_add_watch` system call: + +```console +$ strace --follow-forks --trace='inotify_add_watch' inotifywait --quiet test +inotify_add_watch(3, "test", IN_ACCESS|IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_CLOSE_NOWRITE|IN_OPEN|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF) = 1 +``` + +This shows that the `inotifywait` command is listening for notifications related +to the `test` file. + +## Resolution + +If you encounter the file watcher limit, there are two potential resolutions: +either reduce the number of file watcher registrations, or increase the maximum +file watcher limit. We recommend attempting to reduce the file watcher +registrations first, because increasing the number of file watches may result +in high processor utilization. + +### Reduce watchers + +Many applications include files that change very rarely, and developers often +do not consider these to be part of _their_ application. For example, this may +include third-party dependencies stored in `node_modules`. Tools may watch for +changes to these files and folders, consuming `inotify` watchers. + +These tools typically provide configuration settings to exclude certain files, +paths, and patterns from file watching. For example, Visual Studio Code and +`code-server` apply the following [user workspace setting] by default: + +```json +"files.watcherExclude": { + "**/.git/objects/**": true, + "**/.git/subtree-cache/**": true, + "**/node_modules/**": true, + "**/.hg/store/**": true +}, +``` + +Consider adding other infrequently-changed files to this list, which will cause +Visual Studio Code to poll for changes to those files periodically. For other +software tools, please see the respective user manual. + +[user workspace setting]: https://code.visualstudio.com/docs/getstarted/settings + +### Increase the watch limit + +You can increase the kernel tunable option to increase the maximum number of +`inotify` watches for each user. This is a global setting that applies to all +users sharing the same system, or _Node_ in Kubernetes parlance. To do this, +modify the `sysctl` configuration file, or apply a DaemonSet to the Kubernetes +cluster to apply that change to all nodes automatically. + +For example, create a file called `/etc/sysctl.d/watches.conf` and include the +following contents: + +```text +fs.inotify.max_user_watches = 65536 +``` + +Alternatively, you can use the following DaemonSet with `kubectl apply`: + +```yaml +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: more-fs-watchers + namespace: kube-system + labels: + app: more-fs-watchers + k8s-app: more-fs-watchers +spec: + selector: + matchLabels: + k8s-app: more-fs-watchers + template: + metadata: + labels: + name: more-fs-watchers + k8s-app: more-fs-watchers + annotations: + seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default + apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default + spec: + nodeSelector: + kubernetes.io/os: linux + initContainers: + - name: sysctl + image: alpine:3 + command: + - sysctl + - -w + - fs.inotify.max_user_watches=16384 + resources: + requests: + cpu: 10m + memory: 1Mi + limits: + cpu: 100m + memory: 5Mi + securityContext: + # We need to run as root in a privileged container to modify + # /proc/sys on the host (for sysctl) + runAsUser: 0 + privileged: true + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + containers: + - name: pause + image: k8s.gcr.io/pause:3.5 + command: + - /pause + resources: + requests: + cpu: 10m + memory: 1Mi + limits: + cpu: 100m + memory: 5Mi + securityContext: + runAsNonRoot: true + runAsUser: 65535 + allowPrivilegeEscalation: false + privileged: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + terminationGracePeriodSeconds: 5 +``` + +This DaemonSet will ensure that the corresponding Pod runs on _every_ Linux node +in the cluster. When new nodes join the cluster, such as when an autoscaling +event occurs, the DaemonsSet will ensure that the pod runs on that node as well. +You may delete the DaemonSet by running the following command: + +```console +$ kubectl delete --namespace=kube-system daemonset more-fs-watchers +daemonset.apps "more-fs-watchers" deleted +``` + +However, note that the setting will persist until the node restarts or until +another program sets the `fs.inotify.max_user_watches` setting. + +## See Also + +- Resources for Visual Studio Code and code-server: + - [User and Workspace Settings](https://code.visualstudio.com/docs/getstarted/settings), + in particular, the setting called `files.watcherExclude`. + - [VS Code Setting: files.watcherExclude](https://youtu.be/WMNua0ob6Aw) + (YouTube). + - [My ultimate VSCode + configuration](https://dev.to/vaidhyanathan93/ulitmate-vscode-configuration-4i2o), + a blog post describing a user's preferred settings, including file exclusions. +- [Filesystem notification, part 1: An overview of dnotify and inotify](https://lwn.net/Articles/604686/) + and [Filesystem notification, part 2: A deeper investigation of inotify](https://lwn.net/Articles/605128/) + examine the `inotify` mechanism in detail, along with its predecessor, + `dnotify`. +- Microsoft's Language Server Protocol (LSP) specification describes an approach + for using file watch notifications: + [DidChangeWatchedFiles Notification](https://microsoft.github.io/language-server-protocol/specification#workspace_didChangeWatchedFiles). + Visual Studio Code and code-server, along with many other editors, uses this + protocol for programming language support, and the same constraints and + limitations apply to those tools. +- Kubernetes provides a mechanism for [Setting Sysctls for a Pod](https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#setting-sysctls-for-a-pod). + However, the `inotify` tunable options cannot be set this way, so this is not + applicable to resolving this issue. +- [INotify watch limit](https://blog.passcod.name/2017/jun/25/inotify-watch-limit.html) + provides additional context on the problem and its resolution. +- [inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html), the Linux + manual page related to the `inotify` system call. +- [Kernel Korner - Intro to inotify](https://www.linuxjournal.com/article/8478) diff --git a/manifest.json b/manifest.json index 45e7c2c95..59b4435c6 100644 --- a/manifest.json +++ b/manifest.json @@ -117,7 +117,8 @@ "children": [ { "path": "./guides/admin/resources.md" }, { "path": "./guides/admin/usage-monitoring.md" }, - { "path": "./guides/admin/helm-charts.md" } + { "path": "./guides/admin/helm-charts.md" }, + { "path": "./guides/admin/inotify-watch-limits.md" } ] }, { From ee04d53a256a6c7d53049606378bf081c779bc5b Mon Sep 17 00:00:00 2001 From: Jonathan Yu Date: Mon, 22 Mar 2021 22:19:02 +0000 Subject: [PATCH 2/9] code review comments --- guides/admin/inotify-watch-limits.md | 43 +++++++++++++++------------- 1 file changed, 23 insertions(+), 20 deletions(-) diff --git a/guides/admin/inotify-watch-limits.md b/guides/admin/inotify-watch-limits.md index 4c418832b..6d4bd048e 100644 --- a/guides/admin/inotify-watch-limits.md +++ b/guides/admin/inotify-watch-limits.md @@ -3,17 +3,18 @@ title: Troubleshooting inotify Watcher Limit Problems description: Learn how to resolve problems related to the inotify Watcher Limit. --- -The [`inotify` facility] allows programs to monitor files and directories for -changes, so that they will receive an event immediately whenever a user or -program modifies the file or directory. Many such programs can fall back to -periodically checking files and directories for changes, also known as polling. -Using polling avoids the watch limit, but may result in higher system loading -and a longer interval for the program to detect changes. +The [`inotify` facility] allows programs to monitor files for changes, so that +they will receive an event immediately whenever a user or program modifies the +file. Many such programs can fall back to periodically checking files for +changes, also known as polling. Using polling avoids the watch limit, but may +result in higher system loading and a longer interval for the program to detect +changes. `inotify` requires kernel resources (memory and processor) for each file it tracks. As a result, the Linux kernel limits the number of file watchers that each user can register. The default settings vary according to the host system -distribution; on Ubuntu 20.04 LTS, the default limit is 8,192 watches per instance +distribution; on Ubuntu 20.04 LTS, the default limit is 8,192 watches per +instance. [`inotify` facility]: https://en.wikipedia.org/wiki/Inotify @@ -38,13 +39,13 @@ it is likely that you are encountering this limit. There are three kernel tuning options related to the `inotify` system: -- `fs.inotify.max_queued_events`, which defines an upper bound on the number of - file notification events pending delivery to programs. -- `fs.inotify.max_user_instances`, which determines the maximum number of - `inotify` instances per user. Programs using `inotify` will typically create - a single _instance_, so this limit is unlikely to cause issues. -- `fs.inotify.max_user_watches`, which defines the maximum number of files and - folders that programs can monitor for changes. +- `fs.inotify.max_queued_events`: the upper bound on the number of file + notification events pending delivery to programs. +- `fs.inotify.max_user_instances`: the maximum number of `inotify` instances + per user. Programs using `inotify` will typically create a single _instance_, + so this limit is unlikely to cause issues. +- `fs.inotify.max_user_watches`: the maximum number of files and folders that + programs can monitor for changes. For additional detail regarding the `inotify` system, please see [inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html). @@ -90,8 +91,8 @@ for its usage. [`inotify-consumers`]: https://github.com/fatso83/dotfiles/blob/master/utils/scripts/inotify-consumers -To see the specific files and directories that the tools track for changes, you -can use `strace` to monitor invocations of the `inotify_add_watch` system call: +To see the specific files that the tools track for changes, you can use `strace` +to monitor invocations of the `inotify_add_watch` system call: ```console $ strace --follow-forks --trace='inotify_add_watch' inotifywait --quiet test @@ -104,10 +105,12 @@ to the `test` file. ## Resolution If you encounter the file watcher limit, there are two potential resolutions: -either reduce the number of file watcher registrations, or increase the maximum -file watcher limit. We recommend attempting to reduce the file watcher -registrations first, because increasing the number of file watches may result -in high processor utilization. + +1. Reduce the number of file watcher registrations; or, +1. Increase the maximum file watcher limit. + +We recommend attempting to reduce the file watcher registrations first, because +increasing the number of file watches may result in high processor utilization. ### Reduce watchers From 56294c1a2a51663e074de6944a84405bf1ee6317 Mon Sep 17 00:00:00 2001 From: Katie Horne Date: Tue, 23 Mar 2021 10:58:14 -0500 Subject: [PATCH 3/9] Edit text --- guides/admin/inotify-watch-limits.md | 171 ++++++++++++++------------- 1 file changed, 89 insertions(+), 82 deletions(-) diff --git a/guides/admin/inotify-watch-limits.md b/guides/admin/inotify-watch-limits.md index 6d4bd048e..14c1183f0 100644 --- a/guides/admin/inotify-watch-limits.md +++ b/guides/admin/inotify-watch-limits.md @@ -3,55 +3,53 @@ title: Troubleshooting inotify Watcher Limit Problems description: Learn how to resolve problems related to the inotify Watcher Limit. --- -The [`inotify` facility] allows programs to monitor files for changes, so that -they will receive an event immediately whenever a user or program modifies the -file. Many such programs can fall back to periodically checking files for -changes, also known as polling. Using polling avoids the watch limit, but may -result in higher system loading and a longer interval for the program to detect -changes. - -`inotify` requires kernel resources (memory and processor) for each file it -tracks. As a result, the Linux kernel limits the number of file watchers that -each user can register. The default settings vary according to the host system -distribution; on Ubuntu 20.04 LTS, the default limit is 8,192 watches per -instance. - -[`inotify` facility]: https://en.wikipedia.org/wiki/Inotify - -With some applications and tools, including Webpack or [code-server], you may -encounter an error similar to the following: +When using some applications and tools, including Webpack or [code-server], you +may encounter an error similar to the following: > Watchpack Error (watcher): Error: ENOSPC: System limit for number of file > watchers reached, watch '/some/path' [code-server]: https://github.com/cdr/code-server +This article will show you how to diagnose and troubleshoot this error, which +relates to an elevated number of inotify watchers in use. + +## Background + +[`Inotify`] allows programs to monitor files for changes, so that they receive +an event whenever a user or program modifies a file. `inotify` requires kernel +resources (memory and processor) for each file it tracks. As a result, the Linux +kernel limits the number of file watchers that each user can register. The +default settings vary according to the host system distribution; on Ubuntu 20.04 +LTS, the default limit is 8,192 watches per instance. + +[`Inotify`]: https://en.wikipedia.org/wiki/Inotify + On a 64-bit system, each `inotify` watch that programs register will consume approximately 1 kB of kernel memory, which cannot be swapped to disk and is not counted against the environment memory limit setting. ## Diagnosis -If the total number of watchers is close to the `max_user_watches` setting, then -it is likely that you are encountering this limit. +If you encounter the error that's the focus of this article, the total number of watchers in use is approaching the `max_user_watches` setting. +The following sections will show you how to verify if this is the case. ### Check tunable settings There are three kernel tuning options related to the `inotify` system: -- `fs.inotify.max_queued_events`: the upper bound on the number of file - notification events pending delivery to programs. -- `fs.inotify.max_user_instances`: the maximum number of `inotify` instances +| Option | Description | +| - | - | +| `fs.inotify.max_queued_events` | The upper bound on the number of file + notification events pending delivery to programs | +| `fs.inotify.max_user_instances` | The maximum number of `inotify` instances per user. Programs using `inotify` will typically create a single _instance_, - so this limit is unlikely to cause issues. -- `fs.inotify.max_user_watches`: the maximum number of files and folders that - programs can monitor for changes. + so this limit is unlikely to cause issues | +| `fs.inotify.max_user_watches` | The maximum number of files and folders that + programs can monitor for changes | -For additional detail regarding the `inotify` system, please see -[inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html). - -To see the values for these settings applicable to your environment, run the -following commands: +To see the values for these settings that are applicable to your environment, +run: ```console $ sysctl fs.inotify.{max_queued_events,max_user_instances,max_user_watches} @@ -64,12 +62,17 @@ Because these settings are not namespace-aware, the values will be the same regardless of whether you run the commands on the host system or inside a container running on that host. +> See [inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html) for +additional details regarding the `inotify` system. + ### Identify inotify consumers To identify the programs consuming `inotify` watches, you can use a script that summarizes the information available in the `/proc` filesystem, such as -[`inotify-consumers`]. This script will show the names of programs along with -the number of `inotify` watches registered with the kernel: +[`inotify-consumers`]. + +This script will show the names of programs along with the number of `inotify` +watches registered with the kernel: ```console $ ./inotify-consumers @@ -85,11 +88,12 @@ $ ./inotify-consumers 278 WATCHERS TOTAL COUNT ``` -Please note that this is a third-party script published by an individual who is -not affiliated with Coder, and as such, we cannot provide a warranty or support -for its usage. +> Please note that this is a third-party script published by an individual who +is not affiliated with Coder, and as such, we cannot provide a warranty or +support for its usage. -[`inotify-consumers`]: https://github.com/fatso83/dotfiles/blob/master/utils/scripts/inotify-consumers +[`inotify-consumers`]: +https://github.com/fatso83/dotfiles/blob/master/utils/scripts/inotify-consumers To see the specific files that the tools track for changes, you can use `strace` to monitor invocations of the `inotify_add_watch` system call: @@ -99,29 +103,29 @@ $ strace --follow-forks --trace='inotify_add_watch' inotifywait --quiet test inotify_add_watch(3, "test", IN_ACCESS|IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_CLOSE_NOWRITE|IN_OPEN|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF) = 1 ``` -This shows that the `inotifywait` command is listening for notifications related -to the `test` file. +This example shows that the `inotifywait` command is listening for notifications +related to the `test` file. ## Resolution -If you encounter the file watcher limit, there are two potential resolutions: +If you encounter the file watcher limit, you can: -1. Reduce the number of file watcher registrations; or, -1. Increase the maximum file watcher limit. +1. Reduce the number of file watcher registrations +1. Increase the maximum file watcher limit We recommend attempting to reduce the file watcher registrations first, because increasing the number of file watches may result in high processor utilization. ### Reduce watchers -Many applications include files that change very rarely, and developers often -do not consider these to be part of _their_ application. For example, this may -include third-party dependencies stored in `node_modules`. Tools may watch for -changes to these files and folders, consuming `inotify` watchers. +Many applications include files that change rarely (e.g., third-party +dependencies stored in `node_modules`). Your tools may watch for changes to +these files and folders, consuming `inotify` watchers. These tools typically +provide configuration settings to exclude certain files, paths, and patterns +from file watching. -These tools typically provide configuration settings to exclude certain files, -paths, and patterns from file watching. For example, Visual Studio Code and -`code-server` apply the following [user workspace setting] by default: +For example, Visual Studio Code and `code-server` apply the following [user +workspace setting] by default: ```json "files.watcherExclude": { @@ -133,21 +137,23 @@ paths, and patterns from file watching. For example, Visual Studio Code and ``` Consider adding other infrequently-changed files to this list, which will cause -Visual Studio Code to poll for changes to those files periodically. For other -software tools, please see the respective user manual. +Visual Studio Code to poll (or check periodically) for changes to those files. [user workspace setting]: https://code.visualstudio.com/docs/getstarted/settings +For information on how to do this with other software tools, please see their +documentation/user manuals. + ### Increase the watch limit You can increase the kernel tunable option to increase the maximum number of `inotify` watches for each user. This is a global setting that applies to all -users sharing the same system, or _Node_ in Kubernetes parlance. To do this, -modify the `sysctl` configuration file, or apply a DaemonSet to the Kubernetes -cluster to apply that change to all nodes automatically. +users sharing the same system/Kubernetes node. To do this, modify the `sysctl` +configuration file, or apply a DaemonSet to the Kubernetes cluster to apply that +change to all nodes automatically. -For example, create a file called `/etc/sysctl.d/watches.conf` and include the -following contents: +For example, you can create a file called `/etc/sysctl.d/watches.conf` and +include the following contents: ```text fs.inotify.max_user_watches = 65536 @@ -226,44 +232,45 @@ spec: terminationGracePeriodSeconds: 5 ``` -This DaemonSet will ensure that the corresponding Pod runs on _every_ Linux node -in the cluster. When new nodes join the cluster, such as when an autoscaling -event occurs, the DaemonsSet will ensure that the pod runs on that node as well. -You may delete the DaemonSet by running the following command: +This DaemonSet will ensure that the corresponding pod runs on _every_ Linux node +in the cluster. When new nodes join the cluster, such as during an autoscaling +event, the DaemonsSet will ensure that the pod runs on the new node as well. + +You can delete the DaemonSet by running: ```console $ kubectl delete --namespace=kube-system daemonset more-fs-watchers daemonset.apps "more-fs-watchers" deleted ``` -However, note that the setting will persist until the node restarts or until +However, note that the setting will persist until the node restarts or another program sets the `fs.inotify.max_user_watches` setting. ## See Also +- [INotify watch + limit](https://blog.passcod.name/2017/jun/25/inotify-watch-limit.html) + provides additional context on this problem and its resolution +- [inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html), the Linux + manual page related to the `inotify` system call +- [Kernel Korner - Intro to inotify](https://www.linuxjournal.com/article/8478) +- [Filesystem notification, part 1: An overview of dnotify and + inotify](https://lwn.net/Articles/604686/) and [Filesystem notification, part + 2: A deeper investigation of inotify](https://lwn.net/Articles/605128/) + examine the `inotify` mechanism and its predecessor, `dnotify`, in detail +- Microsoft's Language Server Protocol (LSP) specification [describes an + approach for using file watch + notifications](https://microsoft.github.io/language-server-protocol/specification#workspace_didChangeWatchedFiles) + (Visual Studio Code and code-server, along with many other editors, uses this + protocol for programming language support, and the same constraints and + limitations apply to those tools) - Resources for Visual Studio Code and code-server: - - [User and Workspace Settings](https://code.visualstudio.com/docs/getstarted/settings), - in particular, the setting called `files.watcherExclude`. + - [User and Workspace + Settings](https://code.visualstudio.com/docs/getstarted/settings), in + particular, the setting called `files.watcherExclude` - [VS Code Setting: files.watcherExclude](https://youtu.be/WMNua0ob6Aw) - (YouTube). + (YouTube) - [My ultimate VSCode configuration](https://dev.to/vaidhyanathan93/ulitmate-vscode-configuration-4i2o), - a blog post describing a user's preferred settings, including file exclusions. -- [Filesystem notification, part 1: An overview of dnotify and inotify](https://lwn.net/Articles/604686/) - and [Filesystem notification, part 2: A deeper investigation of inotify](https://lwn.net/Articles/605128/) - examine the `inotify` mechanism in detail, along with its predecessor, - `dnotify`. -- Microsoft's Language Server Protocol (LSP) specification describes an approach - for using file watch notifications: - [DidChangeWatchedFiles Notification](https://microsoft.github.io/language-server-protocol/specification#workspace_didChangeWatchedFiles). - Visual Studio Code and code-server, along with many other editors, uses this - protocol for programming language support, and the same constraints and - limitations apply to those tools. -- Kubernetes provides a mechanism for [Setting Sysctls for a Pod](https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#setting-sysctls-for-a-pod). - However, the `inotify` tunable options cannot be set this way, so this is not - applicable to resolving this issue. -- [INotify watch limit](https://blog.passcod.name/2017/jun/25/inotify-watch-limit.html) - provides additional context on the problem and its resolution. -- [inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html), the Linux - manual page related to the `inotify` system call. -- [Kernel Korner - Intro to inotify](https://www.linuxjournal.com/article/8478) + a blog post describing a user's preferred settings, including file + exclusions From 42509443d63a5b7e6e91b850a8feb7cd2dfab586 Mon Sep 17 00:00:00 2001 From: Katie Horne Date: Tue, 23 Mar 2021 11:14:26 -0500 Subject: [PATCH 4/9] Lint, fix header --- guides/admin/inotify-watch-limits.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/guides/admin/inotify-watch-limits.md b/guides/admin/inotify-watch-limits.md index 14c1183f0..db7c66a79 100644 --- a/guides/admin/inotify-watch-limits.md +++ b/guides/admin/inotify-watch-limits.md @@ -31,8 +31,9 @@ counted against the environment memory limit setting. ## Diagnosis -If you encounter the error that's the focus of this article, the total number of watchers in use is approaching the `max_user_watches` setting. -The following sections will show you how to verify if this is the case. +If you encounter the error that's the focus of this article, the total number of +watchers in use is approaching the `max_user_watches` setting. The following +sections will show you how to verify if this is the case. ### Check tunable settings @@ -243,10 +244,10 @@ $ kubectl delete --namespace=kube-system daemonset more-fs-watchers daemonset.apps "more-fs-watchers" deleted ``` -However, note that the setting will persist until the node restarts or -another program sets the `fs.inotify.max_user_watches` setting. +However, note that the setting will persist until the node restarts or another +program sets the `fs.inotify.max_user_watches` setting. -## See Also +## See also - [INotify watch limit](https://blog.passcod.name/2017/jun/25/inotify-watch-limit.html) From a26b3d6725e385c48d0237a6c99763bfb2bc062a Mon Sep 17 00:00:00 2001 From: Katie Horne Date: Tue, 23 Mar 2021 11:16:50 -0500 Subject: [PATCH 5/9] Use symbol --- guides/admin/inotify-watch-limits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guides/admin/inotify-watch-limits.md b/guides/admin/inotify-watch-limits.md index db7c66a79..6cb803066 100644 --- a/guides/admin/inotify-watch-limits.md +++ b/guides/admin/inotify-watch-limits.md @@ -26,7 +26,7 @@ LTS, the default limit is 8,192 watches per instance. [`Inotify`]: https://en.wikipedia.org/wiki/Inotify On a 64-bit system, each `inotify` watch that programs register will consume -approximately 1 kB of kernel memory, which cannot be swapped to disk and is not +~1 kB of kernel memory, which cannot be swapped to disk and is not counted against the environment memory limit setting. ## Diagnosis From 5cedfb8817f0ce92f7c69549fa16ca863b62bf38 Mon Sep 17 00:00:00 2001 From: Katie Horne Date: Tue, 23 Mar 2021 11:30:40 -0500 Subject: [PATCH 6/9] Remove table --- guides/admin/inotify-watch-limits.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/guides/admin/inotify-watch-limits.md b/guides/admin/inotify-watch-limits.md index 6cb803066..4bae2a999 100644 --- a/guides/admin/inotify-watch-limits.md +++ b/guides/admin/inotify-watch-limits.md @@ -1,5 +1,5 @@ --- -title: Troubleshooting inotify Watcher Limit Problems +title: inotify Watcher Limit Problems description: Learn how to resolve problems related to the inotify Watcher Limit. --- @@ -39,15 +39,13 @@ sections will show you how to verify if this is the case. There are three kernel tuning options related to the `inotify` system: -| Option | Description | -| - | - | -| `fs.inotify.max_queued_events` | The upper bound on the number of file - notification events pending delivery to programs | -| `fs.inotify.max_user_instances` | The maximum number of `inotify` instances - per user. Programs using `inotify` will typically create a single _instance_, - so this limit is unlikely to cause issues | -| `fs.inotify.max_user_watches` | The maximum number of files and folders that - programs can monitor for changes | +- `fs.inotify.max_queued_events`: The upper bound on the number of file + notification events pending delivery to programs +- `fs.inotify.max_user_instances`: The maximum number of `inotify` instances per + user. Programs using `inotify` will typically create a single _instance_, so + this limit is unlikely to cause issues +- `fs.inotify.max_user_watches`: The maximum number of files and folders that + programs can monitor for changes To see the values for these settings that are applicable to your environment, run: From c187cdf1e61ac3db5862b434f6d92effc94c380a Mon Sep 17 00:00:00 2001 From: Katie Horne Date: Tue, 23 Mar 2021 11:35:20 -0500 Subject: [PATCH 7/9] Move to guides/troubleshooting --- guides/troubleshooting/index.md | 6 ++++++ guides/{admin => troubleshooting}/inotify-watch-limits.md | 0 manifest.json | 6 ++++++ 3 files changed, 12 insertions(+) create mode 100644 guides/troubleshooting/index.md rename guides/{admin => troubleshooting}/inotify-watch-limits.md (100%) diff --git a/guides/troubleshooting/index.md b/guides/troubleshooting/index.md new file mode 100644 index 000000000..710a8ee1c --- /dev/null +++ b/guides/troubleshooting/index.md @@ -0,0 +1,6 @@ +--- +title: "Troubleshooting" +description: Learn how to troubleshoot Coder-related errors. +--- + + diff --git a/guides/admin/inotify-watch-limits.md b/guides/troubleshooting/inotify-watch-limits.md similarity index 100% rename from guides/admin/inotify-watch-limits.md rename to guides/troubleshooting/inotify-watch-limits.md diff --git a/manifest.json b/manifest.json index 59b4435c6..6fb3929b3 100644 --- a/manifest.json +++ b/manifest.json @@ -151,6 +151,12 @@ }, { "path": "./guides/api.md", "navigable": false + }, + { "path": "./guides/troubleshooting/index.md", + "navigable": false, + "children": [ + { "path": "./guides/troubleshooting/inotify-watch-limits.md" } + ] } ] }, From 26e5a3c895132e70a1f6a0ae64acc0e43df836f2 Mon Sep 17 00:00:00 2001 From: Katie Horne Date: Tue, 23 Mar 2021 11:41:20 -0500 Subject: [PATCH 8/9] Remove extra manifest entry --- manifest.json | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/manifest.json b/manifest.json index 6fb3929b3..68286e091 100644 --- a/manifest.json +++ b/manifest.json @@ -117,8 +117,7 @@ "children": [ { "path": "./guides/admin/resources.md" }, { "path": "./guides/admin/usage-monitoring.md" }, - { "path": "./guides/admin/helm-charts.md" }, - { "path": "./guides/admin/inotify-watch-limits.md" } + { "path": "./guides/admin/helm-charts.md" } ] }, { From 1d004f538b113bc2eff87ad37b676bac362e0b00 Mon Sep 17 00:00:00 2001 From: Katie Horne Date: Tue, 23 Mar 2021 12:41:56 -0500 Subject: [PATCH 9/9] Apply changes from code review --- guides/troubleshooting/inotify-watch-limits.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/guides/troubleshooting/inotify-watch-limits.md b/guides/troubleshooting/inotify-watch-limits.md index 4bae2a999..4d2e3c471 100644 --- a/guides/troubleshooting/inotify-watch-limits.md +++ b/guides/troubleshooting/inotify-watch-limits.md @@ -16,14 +16,14 @@ relates to an elevated number of inotify watchers in use. ## Background -[`Inotify`] allows programs to monitor files for changes, so that they receive +[`inotify`] allows programs to monitor files for changes, so that they receive an event whenever a user or program modifies a file. `inotify` requires kernel resources (memory and processor) for each file it tracks. As a result, the Linux kernel limits the number of file watchers that each user can register. The default settings vary according to the host system distribution; on Ubuntu 20.04 LTS, the default limit is 8,192 watches per instance. -[`Inotify`]: https://en.wikipedia.org/wiki/Inotify +[`inotify`]: https://en.wikipedia.org/wiki/Inotify On a 64-bit system, each `inotify` watch that programs register will consume ~1 kB of kernel memory, which cannot be swapped to disk and is not @@ -42,8 +42,8 @@ There are three kernel tuning options related to the `inotify` system: - `fs.inotify.max_queued_events`: The upper bound on the number of file notification events pending delivery to programs - `fs.inotify.max_user_instances`: The maximum number of `inotify` instances per - user. Programs using `inotify` will typically create a single _instance_, so - this limit is unlikely to cause issues + user (programs using `inotify` will typically create a single _instance_, so + this limit is unlikely to cause issues) - `fs.inotify.max_user_watches`: The maximum number of files and folders that programs can monitor for changes @@ -62,7 +62,7 @@ regardless of whether you run the commands on the host system or inside a container running on that host. > See [inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html) for -additional details regarding the `inotify` system. +> additional details regarding the `inotify` system. ### Identify inotify consumers @@ -107,7 +107,7 @@ related to the `test` file. ## Resolution -If you encounter the file watcher limit, you can: +If you encounter the file watcher limit, you can do one of two things: 1. Reduce the number of file watcher registrations 1. Increase the maximum file watcher limit @@ -233,7 +233,7 @@ spec: This DaemonSet will ensure that the corresponding pod runs on _every_ Linux node in the cluster. When new nodes join the cluster, such as during an autoscaling -event, the DaemonsSet will ensure that the pod runs on the new node as well. +event, the DaemonSet will ensure that the pod runs on the new node as well. You can delete the DaemonSet by running: