scylla does not start when kernel inotify limits are exceeded

[avikivity]

Each tls instance consumes an inotify watch, and there can be multiple tls instances per shard. A large machine can run out, and will fail startup. The default is 128, which is enough for 64 shards.

[psarna]

Maybe this error should also not be fatal, but instead print an error message in the logs? I guess it depends if the mechanisms that rely on inotify also allow other ways of reloading observed files (e.g. via a signal or REST or whatever).

[avikivity]

@psarna it threw an exception, but the exception was converted to an assert() when a sharded<> instance was destroyed incorrectly. @elcallio promised to fix that.

[avikivity]

Backported to 4.1, 4.2, 4.3.

[elcallio]

Question: Should we try to address this on a seastar level? While the basic problem or shard multiplication cannot be solved, we could maybe fix it somewhat for the usage pattern of shard-shared credentials builder generating a reloadable credentials object per shard.
With some (terrible) juggling of foreign pointers it should be possible to make only a single shard actually use inotify, and the rest cross-shard subscribe to the originating shards notifications:

There will be a lot of cross-shard calls when stuff changes, but...

We can also add a fallback option for the actual originating shard reloader, to use polling iff inotify is not available.

[avikivity]

We could, but it's a huge amount of work compared to writing to a sysctl file.

[elcallio]

I take that as a down prioritization of the idea.

[avikivity]

Yes.