snapcraft/commands/daemon.start: bump fs.inotify.max_user_watches

[simondeziel]

https://documentation.ubuntu.com/lxd/en/latest/reference/server_settings/#etc-sysctl-conf recommends setting fs.inotify.max_user_watches to 10485761 for production setup. Since we already set fs.inotify.max_user_instances to 1024 it means we expect a given host to at least accomodate for that many containers. However, launching ~85 containers apparently pushed LXD into consuming all the user watches:

$ for i in $(seq 100); do lxc launch ... ; done
$ sudo systemctl reload snap.lxd.daemon
Failed to allocate directory watch: Too many open files

[tomponline]

Any risk to doing this by default in the snap?

Any thoughts on why only the existing setting was included by default?

[tomponline]

Are the inotify entries being used caused by lxd or the guest os itself?

[mihalicyn]

Are the inotify entries being used caused by lxd or the guest os itself?

AFAIU, by LXD itself (device monitor uses inotify to watch directories recursively).

[tomponline]

AFAIU, by LXD itself (device monitor uses inotify to watch directories recursively).

It'd be good to get an understanding of why number of watchers in LXD are associated to number of instances.
Also I would've expected LXD on a modern system to be using fanotify, so doesn't need to setup recursive watchers AFAIK.

Or perhaps this isn't coming from LXD, but instead watchers from the guest OS?

[mihalicyn]

Also I would've expected LXD on a modern system to be using fanotify, so doesn't need to setup recursive watchers AFAIK.

yes, this counter is about number of watches which are created with syscall.InotifyAddWatch.

[mihalicyn]

I have tried to reproduce this behavior on my machine and have created more than 100 containers.

I haven't received any errors from LXD daemon itself (and it's definitely using fanotify at least on my Ubuntu 22.04).

But, interesting thing I noticed is that I can't start more than ~97 containers. lxc start <..> always succeeds but container stops just right after it's started.
When I started investigating this I found:

$ cat /var/snap/lxd/common/lxd/logs/tops-puma/console.log
systemd 249.11-0ubuntu3.12 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 22.04.4 LTS!

Failed to create control group inotify object: Too many open files
Failed to allocate manager object: Too many open files
[ESC[0;1;31m!!!!!!ESC[0m] Failed to allocate manager object.
Exiting PID 1...
(END)

[tomponline]

I have tried to reproduce this behavior on my machine and have created more than 100 containers.

I haven't received any errors from LXD daemon itself (and it's definitely using fanotify at least on my Ubuntu 22.04).

But, interesting thing I noticed is that I can't start more than ~97 containers. lxc start <..> always succeeds but container stops just right after it's started. When I started investigating this I found:

$ cat /var/snap/lxd/common/lxd/logs/tops-puma/console.log
systemd 249.11-0ubuntu3.12 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 22.04.4 LTS!

Failed to create control group inotify object: Too many open files
Failed to allocate manager object: Too many open files
[ESC[0;1;31m!!!!!!ESC[0m] Failed to allocate manager object.
Exiting PID 1...
(END)

Ah so its not LXD using them but the containers themselves.

[tomponline]

@simondeziel please can you backport this to latest-candidate, 5.21-edge, 5.21-candidate, 5.0-edge thanks