Framework for writing Out-of-Tree Kernel Modules

[CMCDragonkai]

From inspecting the netfilter code, we may need to write custom kernel modules to achieve some of our Relay or Emergence goals.

So here I've sketched out a skeleton for writing kernel modules. It is based on the Wireguard software.

The wireguard-tools is a package containing userspace tools which relies on the wireguard kernel module. In most packaging systems, these are separated. We would do something similar. However because wireguard is out-of-tree, the source code for both the userspace code and kernel space code is in the same location. Which makes it easier for us. That code is here: https://github.com/WireGuard/WireGuard/tree/master/src

In NixOS, the userspace tools derivation is here: https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/networking/wireguard-tools/default.nix while the kernel module derivation which uses wireguard-tools as a dependency (but only for it derivation attributes, not as a package dependency (remember this is important!)) is here: https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/wireguard/default.nix

Finally the wireguard kernel module is written using the DKMS framework. https://github.com/dell/dkms It appears to allow kernel modules to be recompiled when a new kernel is installed. I don't know if this is relevant when using NixOS, since we don't have this sort of automatic recompilation happening except via explicitly through the Nix evaluation system.

[CMCDragonkai]

I'm assigning myself as well, as I need to show @ramwan how to use nix-shell and Nix to setup a relevant development environment for Haskell, C, Go and Kernel Development.

[CMCDragonkai]

We went through an example of creating a shell.nix for Haskell/Stack + Go development, and worked through an example of binding to C code, either from the standard library or via a shared object.

Note that Haskell's RTS enables GHCi runtime linking. You have to use stack ghci --ghci-options -lLIB.so. Note that will always look for libLIB.so in the current directory. No idea why it adds lib as prefix at the moment.

Shared object linking is fine, but I wish to get some documentation on static linking and linking configured by stack instead of using ghc and stack individually. @mokuki082 can you help?

[ramwan]

Notes on enabling Wireguard and IPVS on nixos:

• add the following line to /etc/nixos/configuration.nix
  boot.extraModulePackages = [ config.boot.kernelPackages.wireguard ];
• add wireguard to your profile's config.nix
• reboot your system
• add "ip_vs" to the boot.kernelModules parameter in /etc/nixos/configuration.nix

[CMCDragonkai]

IPVS is an "in-tree" kernel module, that's why it can just be enabled using modprobe (imperatively), or via boot.kernelModules declaratively. Whereas wireguard is still an "out-of-tree" kernel module, which is why it had to be referenced using boot.extraModulePackages. Note that in the master branch of Nixpkgs, the the userspace tools for wireguard change to pkgs.wireguard-tools, whereas the kernel module stays as linuxPackages.wireguard.

Also the config.boot.kernelPackages is using the config fixed point to refer to boot.kernelPackages which is set to linuxPackages.

Later when wireguard gets merged and becomes an in-tree kernel module, you can just load it via boot.kernelModules. It uses DKMS framework to create this. Check out the pkgs/os-specific/linux/wireguard/default.nix to see how to write a custom derivation for the kernel module in NixOS.

[CMCDragonkai]

It's important to understand this: https://stackoverflow.com/questions/22891705/whats-the-difference-between-insmod-and-modprobe

Basically use modinfo on any compiled kernel module which will give you information such as what that kernel module depends on.

If you try to use insmod on a compiled kernel module in the current directory, it may result in errors about symbols not being found. That means its dependencies were not resolved. In those cases, using modinfo, you can then use modprobe to load any dependencies ahead of time, and then run insmod on your current kernel module.

Another thing is that the linuxPackages.kernel version must be the same version as your current OS's kernel. Check your configuration.nix for the linuxPackages.kernel.version (easiest is to use <nixpkgs> when entering the shell). Once that's done, you should know that linuxPackages.kernel.dev is an derivation output just like .out that is derived from the same linuxPackages.kernel drv. You can see this happening in nixpkgs/pkgs/os-specific/linux/kernel/manual-config.nix and look for the postInstall attribute. You'll see how it constructs the $dev directory contents. The reason it does this is so that it simulates what the Linux kernel module compilation system expects. It expects a directory usually situated in /lib/modules/... in a normal Linux FHS that contains headers that we can include. However on NixOS, this doesn't exist. This is why the NixOS manual section on compiling your own kernel module (https://nixos.org/nixos/manual/index.html#sec-linux-config-developing-modules) asks you to enter the shell and use the $dev environment variable. Remember that all attributes in a derivation become environment variables in the build shell.

Note that you don't actually need to run: nix-build '<nixpkgs>' -A linuxPackages.kernel.dev. Because that's redundant, as you can just use $dev environment variable to refer to the store path containing the kernel module headers.