models: add new setting to auto-load kernel modules

[foersleo]

Issue number: #3050, #2409

Description of changes:

Loading specific kernel modules can be necessary to use specific features on your node. One example would be loading the correct module for your choice of scheduling algorithm for ipvs.

Loading kernel modules on boot currently requires the use and maintenance of an init-container image to load the desired kernel modules on boot. We can simplify that by adding the additional setting load-on-boot for kernel modules and utilizing the already available systemd-modules-load service.

Module auto-loading is in conflict with blocking modules from loading through the sibling setting allowed. Hence, do not auto-load a module on boot if the module is not allowed at the same time. Blocking takes precedence, as it is the more prohibitive operation.

Testing done:

I have run tests with some variations of module specific options on a local vm in my dev-system and results were as expected:

• autoloading and allowing the module to be loaded:

  [settings.kernel.modules.ip_vs_lc]
  allowed = true
  load-on-boot = true
  

  Results in autoloading the required module and dependency modules when re-triggering systemd-modules-load during boot:

  [  OK  ] Stopped Load Kernel Modules.
           Stopping Load Kernel Modules...
           Starting Load Kernel Modules...
  [    5.244013] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
  [    5.244557] IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
  [    5.245233] IPVS: ipvs loaded.
  [    5.250915] IPVS: [lc] scheduler registered.
  [  OK  ] Finished Load Kernel Modules.
  [...]
  bash-5.1# lsmod | grep ip_vs
  ip_vs_lc               16384  0
  ip_vs                 192512  2 ip_vs_lc
  bash-5.1# cat /etc/modules-load.d/modules-load.conf 
  ip_vs_lc
  

• conflict of blocking and auto-loading does not add the module to the config file and does not load the module:

  [settings.kernel.modules.ip_vs_lc]
  allowed = false
  load-on-boot = true
  

  Results in not autoloading the and not adding it to the config file:

  [  OK  ] Stopped Load Kernel Modules.
           Stopping Load Kernel Modules...
           Starting Load Kernel Modules...
  [  OK  ] Finished Load Kernel Modules.
  [...]
  bash-5.1# lsmod | grep ip_vs
  bash-5.1# cat /etc/modules-load.d/modules-load.conf
  

• Accidental overriding of a blocking setting is ruled out by toml not allowing to define the same table twice. The userdata configuration system will be derailed with a parser error for providing invalid user-data.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[stmcginnis]

Awesome, this looks like a big usability improvement for anyone needing to load additional modules.

With the new setting, you will need to add a migration to add/remove that setting on upgrade/downgrade. There is an example of doing something similar here: ac1e6e9

Other than the missing migration, this looks great!

[foersleo]

I will have to have a look at the migrations part. I have not done that before.

[foersleo]

⬆️ tended the renaming to autoload as pointed out by Markus.

I will add the migrations part next week as this seems to be a little bit more specialized with a field in a variably named table. Will have to do some reading.

[foersleo]

⬆️ Added the migrations and bumped the Release version before, now that 1.15.0 is released and these migrations will be relevant when moving to the next version.

I ended up needing 4 separate migrations. I was contemplating if it would make more sense to have one big custom migration or not, but maybe those with more experience writing migrations can chime in here. I was also wondering if it would be possible to do multiple migrate calls the run function of one migration and combine their results, although I did not find precedent for that in existing migrations. So probably it is prohibitive to do that? I would be interested in getting some insight on that from people that know more about the migrations system.

I did test the migrations, the following is the gist of what happened:

Upgrade:

[ec2-user@admin]$ sudo sheltie
bash-5.1# updog update -i 1.16.0 -r -n
Starting update to 1.16.0
Reboot scheduled for Tue 2023-09-19 09:58:57 UTC, use 'shutdown -c' to cancel.
Update applied: aws-k8s-1.27 1.16.0
[...]
[ec2-user@admin]$ sudo sheltie
bash-5.1# cd /var/lib/bottlerocket/datastore/current/live/
bash-5.1# cd settings/kernel/
bash-5.1# cat modules.affected-services 
["kernel-modules"]
bash-5.1# cd ../../
bash-5.1# cd configuration-files/
bash-5.1# cat modules-load/path 
"/etc/modules-load.d/modules-load.conf"
bash-5.1# cat modules-load/template-path 
"/usr/share/templates/modules-load"
bash-5.1# cd ../
bash-5.1# cd services/kernel-modules/
bash-5.1# cat configuration-files 
["modprobe-conf","modules-load"]
bash-5.1# cat restart-commands 
["/usr/bin/systemctl restart systemd-modules-load"]

Downgrade (setup one module for autoload previously to see that these keys get cleaned up correctly as well):

bash-5.1# apiclient set kernel.modules.ip_vs_lc.allowed=true kernel.modules.ip_vs_lc.autoload=true
bash-5.1# cd settings/kernel/modules
bash-5.1# ls
ip_vs_lc
bash-5.1# cd ip_vs_lc/
bash-5.1# ls
allowed  autoload
bash-5.1# cat *
true
true
bash-5.1# lsmod | grep ip_vs
ip_vs_lc               16384  0
ip_vs                 192512  2 ip_vs_lc
bash-5.1# signpost rollback-to-inactive
bash-5.1# reboot
[...]
bash-5.1# cd /var/lib/bottlerocket/datastore/current/live/
bash-5.1# cd settings/kernel/modules/ip_vs_lc/ 
bash-5.1# ls
allowed
bash-5.1# lsmod | grep ip_vs
bash-5.1# cd ../../../../
bash-5.1# cd services/kernel-modules/
bash-5.1# cat configuration-files 
["modprobe-conf"]
bash-5.1# cat restart-commands 
[]
bash-5.1# cd ../../
bash-5.1# cd configuration-files/
bash-5.1# ls modules-load
ls: cannot access 'modules-load': No such file or directory

[foersleo]

⬆️ force push fixed some formatting issues.

[foersleo]

Last force pushed fixed additional clippy warnings.

[bcressey]

One example would be loading the correct module for your choice of scheduling algorithm for ipvs.

Do we have other examples? I noted in #3391 that the kernel seems to auto-load the IPVS modules.

[bcressey]

Quite the crash course in migrations, but they look OK to me.

[foersleo]

One example would be loading the correct module for your choice of scheduling algorithm for ipvs.

Do we have other examples? I noted in #3391 that the kernel seems to auto-load the IPVS modules.

I missed the part that IPVS modules are now actually working to be autoloaded. I remember back when I was looking at the original ticket that the overhaul of the kube-proxy code indicated that starting in k8s 1.27 the modules could be loaded automatically. But I did not revisit this until now.

Another instance of people loading modules manually would be br_netfilter as raised in #3052 . But that again seems to be just a different facet of the same "kube-proxy needs different modules for different settings".

Even if the autoloading of ipvs modules works with k8s-1.27 plus, this is probably a good change for people still using older versions and those use-cases we do not know about for comparably low cost.

[foersleo]

⬆️ force-push changed the restart-command to use try-restart instead of restart as suggested by Ben.

[markusboehme]

Nice work, especially with the migrations--learned a thing or two! 🧙

[stmcginnis]

🎉