chore: add doc on increasing inotify-watch-limits

[khorne3]

For CH#6016

[bpmct]

As we get the ability to group guides together, could we put this into something like troubleshooting?

[khorne3]

As we get the ability to group guides together, could we put this into something like troubleshooting?

@bpmct Yep! I spun up a working preview so I'll add a troubleshooting section too

[jawnsy]

I don't think this actually helps in this case, because these sysctls are not namespaced and therefore not on the allow list

[jawnsy]

Suggest titling this "Increasing the inotify Watch Limit" or "Increasing the File Watcher Limit" (I think inotify_add_watch is typically styled in lowercase as it's the name of a system call)

Suggested change

	title: Inotify Watch Limit Increase
	title: Increasing the inotify Watch Limit

More generally, though, it seems we may want to have this be part of a larger page about tuning options (or more specifically, kernel tuning options) for Coder?

[jawnsy]

This is not Webpack-specific, so I would suggest making this more generic.

Suggested change

	When running a webpack project, you may encounter an error similar to the
	With some applications and tools, including Webpack or code-server, you may encounter an error similar to the

[jawnsy]

Suggest adding more info about what the inotify mechanism is for, what it does, and the impact of changing the watch limit.

Suggested change

This results from a low number of [inotify

The [inotify facility](https://en.wikipedia.org/wiki/Inotify) allows programs to monitor files and directories for changes, so that they will receive an event immediately whenever a user or program modifies the file or directory. Because the inotify mechanism requires some kernel resources (memory and processor) per watched file, there is a limit on the number of registered file watchers. For large projects with many files and folders, programs may hit this limit, resulting in the above error message.

[jawnsy]

Suggested change

	Increase the number of file watchers. Because the setting quantifying the number
	If you encounter the file watcher limit, there are two potential resolutions: either reduce the number of file watcher registrations, or increase the maximum file watcher limit. We recommend attempting to reduce the file watcher registrations first, because increasing the number of file watches may result in high processor utilization.
	For code-server, you may exclude some files, such as those belonging to third-party project dependencies, by modifying the `files.watcherExclude` setting. Please see the user guides for the software registering the watches to learn more.

[jawnsy]

Suggested change

	One way to do this is to use a daemonset in the cluster with a privileged
	One way to do this is to use a DaemonSet in the cluster with a privileged

DaemonSet is a proper noun so should be capitalized, and we should stylize it as indicated in the official Kubernetes documentation

[jawnsy]

@scsmithr How come this uses sh -c instead of directly running the sysctl command?

[jawnsy]

I tested this and it works - there's only so much we can do since we need a privileged container to access /proc/sys, but I at least tried to cut down on the host namespaces we were sharing into the DaemonSet pods:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: more-fs-watchers
  namespace: kube-system
  labels:
    app: more-fs-watchers
    k8s-app: more-fs-watchers
spec:
  selector:
    matchLabels:
      k8s-app: more-fs-watchers
  template:
    metadata:
      labels:
        name: more-fs-watchers
        k8s-app: more-fs-watchers
      annotations:
        seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
        apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      initContainers:
        - name: sysctl
          image: alpine:3
          command:
            - sysctl
            - -w
            - fs.inotify.max_user_watches=16384
          resources:
            requests:
              cpu: 10m
              memory: 1Mi
            limits:
              cpu: 100m
              memory: 5Mi
          securityContext:
            # We need to run as root in a privileged container to modify
            # /proc/sys on the host (for sysctl)
            runAsUser: 0
            privileged: true
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
      containers:
        - name: pause
          image: k8s.gcr.io/pause:3.5
          command:
            - /pause
          resources:
            requests:
              cpu: 10m
              memory: 1Mi
            limits:
              cpu: 100m
              memory: 5Mi
          securityContext:
            runAsNonRoot: true
            runAsUser: 65535
            allowPrivilegeEscalation: false
            privileged: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
      terminationGracePeriodSeconds: 5

[jawnsy]

@scsmithr we should set requests/limits too. The pod should be set with a read-only rootfs for security. And I suggest we use the sleep command instead of tail -f?

I am actually not sure if we need privileges other than the ability to write to sysfs? We should check this...

[jawnsy]

Suggested change

	## Helpful Resources
	## See Also

We can't guarantee these will be helpful 😉

[jawnsy]

@scsmithr Do these really need to share host namespaces? I don't believe so, since sysfs is being mounted read-write. We shouldn't recommend things that don't meet Kubernetes' security recommendations in our docs

Especially since this is privileged and running as a DaemonSet, we should try to restrict things as much as possible: https://kubernetes.io/docs/concepts/security/pod-security-standards/

[jawnsy]

I've opened #183 and will work on this there