[linux-hardened] RANDSTRUCT can't be enabled, issues with out of tree modules

[delroth]

We've had to disable the RANDSTRUCT kernel GCC plugin due to issues with out-of-tree modules. It seems like the RANDSTRUCT seed that gets picked is different for kernel builds vs. module builds, causing module loading to fail:

Jan 06 05:31:02 yew.delroth.net kernel: wireguard: version magic '4.20.0 SMP mod_unload modversions RANDSTRUCT_PLUGIN_844e5ec4998ce0aa52b0d78fee70e491e939d7be14b88eb3af6184ed5d811399' should be '4.20.0 SMP mod_unload modversions RANDSTRUCT_PLUGIN_76391e9f4a29959bd84176fb339a135e386d12fa3bcfe5032139107ca71ba9a3'

Filing this to track a fix so we can later on re-enable RANDSTRUCT.

[delroth]

So as it turns out this is "by design": see the last few paragraphs in https://lwn.net/Articles/722293/

If we want to ship RANDSTRUCT in nixpkgs, we'll need to have a fixed seed for all nixpkgs users, which reduces the usefulness of RANDSTRUCT (it doesn't make it useless, since it still increases diversity and thus makes exploits harder to develop).

NixOS could however allow users to define their own seed -- at the cost of not being able to use binary caches for kernel / module packages. It's not perfect -- if build machine == deployment machine the seed will be trivially readable in the Nix store, but the kernel itself is readable in the Nix store and I'm pretty sure from there one can recover the seed anyway. This still forces attackers to specifically target NixOS-hardened setups, which I imagine is a population below the caring threshold :-)

Plan:

1. Extend our kernel / module builds to be able to inject a RANDSTRUCT seed.
2. Define a fixed default seed for NixOS.
3. Add a NixOS option to override the seed, with proper documentation as to the security and usability tradeoffs (no shared binary caching,

Not completely sure how to implement this just yet, just throwing it out here for potential feedback.

[joachifm]

Any mitigation strategy that relies on compile-time randomization is weakened/rendered useless by distributing the kernel binaries, even if the seed is not leaked directly, because you can see all the compile-time offsets & so on regardless of how they were determined (randomly or not). This is not specific to randstruct but affects e.g., KASLR which we also enable in the distro kernel.

It seems to me that allowing the seed to vary between builds is somewhat stronger than using a fixed seed for all builds. A local seed is strongest, obviously, subject to the tradeoffs you've listed.

[delroth]

Agreed. I'm planning to use something like SHA256(kernel version || "user configured seed (with nixpkgs default)"). Does that seem fine with you?

[joachifm]

I'd be happy to take anything that allows us to enable the feature and have out-of-tree modules work. That said, some half-baked ideas below.

use something like SHA256(kernel version || "user configured seed (with nixpkgs default)")

I take it that the reason to index on the kernel version is to ensure that the value changes intermittently even with a fixed input seed? If so it seems like kernel version strings are too predictable to offer much improvement. Maybe the nixpkgs default could be derived from the nixpkgs revision (or maybe use that as the index). Hashing the input to get a predictably sized value seems like a good idea in any case. I suspect that for distro kernels it won't matter too much what we do for the seed.

My inclination was to try to carry over the build-time random seed to the out-of-tree module build somehow. If carrying the seed forward works, then there'd be no reason to let the user override the seed as such because it'll be different every time a build runs (and would be shared by all the packages built as part of the linuxPackages set). I think we could simply embed a string like "local" in the drv name to ensure local builds.

Given that kernel packages are rebuilt more often than new kernel versions are released, the range should be a fair bit greater with build-time random values (assuming they don't know exactly which channel/rev you're running on), but I've no idea whether that translates to significant increase in dollar cost to carry out an exploit. It seems most prudent to just assume that compile-time randomization in distro kernels is trivially defeatible & focus on making the local build scenario as good as possible.

For reproducibility, making the seed explicit/exogenous would be good, so that's something to consider & favours your approach over build-time randomization.

EDIT: I should note that I have no particular expertise in crypto, so take all of the above with your preferred condiments, or disregard entirely.

[delroth]

The seed is already being carried over:

[nix-shell:/tmp/source/src]$ cat /nix/store/78r80qai3vy34ljhnic6p5yrg60hvs0x-linux-4.20-dev/lib/modules/4.20.0/build/scripts/gcc-plugins/randomize_layout_seed.h 
const char *randstruct_seed = "0c7d0bf9e833da5133ea89a8ccc859a297256f3ef4d8d96861fb07e2fb561942";

The problem is that since it's being randomly generated during kernel build time, two kernels with the same derivation hash can end up with two different seeds. Not only is this fundamentaly broken, but that means that all out-of-tree module builds are then "contaminated" by the same non-reproducibility problem: two modules with the same derivation hash ended up being built with two different RANDSTRUCT seeds, causing them to be incompatible with other kernel builds.

I think the only way this can be solved is by having a fixed seed part of the kernel derivation. I'm a complete Nix noob so I'd love to be proven wrong though!

[joachifm]

What I had in mind is that if the seed is captured when the kernel is initially built, all you should need to do is rig the out-of-tree kernel module build to use that captured seed. It should then not matter that the seed is recreated whenever a build of the kernel is triggered, as you presumably build all of linuxPackages against the same kernel build (and hence the same captured seed). This is quite rigid, however, if you for whatever reason cause the kernel drv to become invalid and so need to be rebuilt, the seed would change and you would be unable to load new modules without rebooting.

[joachifm]

I had really expected this to Just Work, so there may be a wrinkle in there that I've failed to consider. In any case, I think an externally provided seed is the way to go, even if it should be possible to fix the build process to work the way I've outlined.

[joachifm]

It also fails obviously if it is not the case that all of linuxPackages was built against the exact same kernel (presumably on the same machine, at the same time). Just a Bad Idea all round, really.

[symphorien]

The seed could be (deterministically) derived from the store path of the kernel source (or some other kernel related derivation). This fulfills reproducibility and some amount of diversity.

[delroth]

With #53802 merged, RANDSTRUCT is now re-enabled with a seed that's derived from kernel source + config file Nix store path. Let's see if it sticks this time :-)

I'll repurpose this issue for the next step of the plan described earlier: making the seed user-configurable, and adding the required NixOS plumbing to allow users to set their own for their deployments.

[joachifm]

I've merged @delroth's patch. Thank you.