Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[linux-hardened] RANDSTRUCT can't be enabled, issues with out of tree modules #53592

Closed
delroth opened this issue Jan 7, 2019 · 11 comments

Comments

Projects
None yet
3 participants
@delroth
Copy link
Contributor

commented Jan 7, 2019

We've had to disable the RANDSTRUCT kernel GCC plugin due to issues with out-of-tree modules. It seems like the RANDSTRUCT seed that gets picked is different for kernel builds vs. module builds, causing module loading to fail:

Jan 06 05:31:02 yew.delroth.net kernel: wireguard: version magic '4.20.0 SMP mod_unload modversions RANDSTRUCT_PLUGIN_844e5ec4998ce0aa52b0d78fee70e491e939d7be14b88eb3af6184ed5d811399' should be '4.20.0 SMP mod_unload modversions RANDSTRUCT_PLUGIN_76391e9f4a29959bd84176fb339a135e386d12fa3bcfe5032139107ca71ba9a3'

Filing this to track a fix so we can later on re-enable RANDSTRUCT.

@delroth delroth changed the title [linux-hardened] RANDSTRUCT causes issues with out of tree modules [linux-hardened] RANDSTRUCT can't be enabled, issues with out of tree modules Jan 7, 2019

delroth referenced this issue Jan 7, 2019

@delroth

This comment has been minimized.

Copy link
Contributor Author

commented Jan 8, 2019

So as it turns out this is "by design": see the last few paragraphs in https://lwn.net/Articles/722293/

If we want to ship RANDSTRUCT in nixpkgs, we'll need to have a fixed seed for all nixpkgs users, which reduces the usefulness of RANDSTRUCT (it doesn't make it useless, since it still increases diversity and thus makes exploits harder to develop).

NixOS could however allow users to define their own seed -- at the cost of not being able to use binary caches for kernel / module packages. It's not perfect -- if build machine == deployment machine the seed will be trivially readable in the Nix store, but the kernel itself is readable in the Nix store and I'm pretty sure from there one can recover the seed anyway. This still forces attackers to specifically target NixOS-hardened setups, which I imagine is a population below the caring threshold :-)

Plan:

  1. Extend our kernel / module builds to be able to inject a RANDSTRUCT seed.
  2. Define a fixed default seed for NixOS.
  3. Add a NixOS option to override the seed, with proper documentation as to the security and usability tradeoffs (no shared binary caching,

Not completely sure how to implement this just yet, just throwing it out here for potential feedback.

@joachifm

This comment has been minimized.

Copy link
Contributor

commented Jan 8, 2019

Any mitigation strategy that relies on compile-time randomization is weakened/rendered useless by distributing the kernel binaries, even if the seed is not leaked directly, because you can see all the compile-time offsets & so on regardless of how they were determined (randomly or not). This is not specific to randstruct but affects e.g., KASLR which we also enable in the distro kernel.

It seems to me that allowing the seed to vary between builds is somewhat stronger than using a fixed seed for all builds. A local seed is strongest, obviously, subject to the tradeoffs you've listed.

@delroth

This comment has been minimized.

Copy link
Contributor Author

commented Jan 8, 2019

Agreed. I'm planning to use something like SHA256(kernel version || "user configured seed (with nixpkgs default)"). Does that seem fine with you?

@joachifm

This comment has been minimized.

Copy link
Contributor

commented Jan 8, 2019

I'd be happy to take anything that allows us to enable the feature and have out-of-tree modules work. That said, some half-baked ideas below.

use something like SHA256(kernel version || "user configured seed (with nixpkgs default)")

I take it that the reason to index on the kernel version is to ensure that the value changes intermittently even with a fixed input seed? If so it seems like kernel version strings are too predictable to offer much improvement. Maybe the nixpkgs default could be derived from the nixpkgs revision (or maybe use that as the index). Hashing the input to get a predictably sized value seems like a good idea in any case. I suspect that for distro kernels it won't matter too much what we do for the seed.

My inclination was to try to carry over the build-time random seed to the out-of-tree module build somehow. If carrying the seed forward works, then there'd be no reason to let the user override the seed as such because it'll be different every time a build runs (and would be shared by all the packages built as part of the linuxPackages set). I think we could simply embed a string like "local" in the drv name to ensure local builds.

Given that kernel packages are rebuilt more often than new kernel versions are released, the range should be a fair bit greater with build-time random values (assuming they don't know exactly which channel/rev you're running on), but I've no idea whether that translates to significant increase in dollar cost to carry out an exploit. It seems most prudent to just assume that compile-time randomization in distro kernels is trivially defeatible & focus on making the local build scenario as good as possible.

For reproducibility, making the seed explicit/exogenous would be good, so that's something to consider & favours your approach over build-time randomization.

EDIT: I should note that I have no particular expertise in crypto, so take all of the above with your preferred condiments, or disregard entirely.

@delroth

This comment has been minimized.

Copy link
Contributor Author

commented Jan 8, 2019

The seed is already being carried over:

[nix-shell:/tmp/source/src]$ cat /nix/store/78r80qai3vy34ljhnic6p5yrg60hvs0x-linux-4.20-dev/lib/modules/4.20.0/build/scripts/gcc-plugins/randomize_layout_seed.h 
const char *randstruct_seed = "0c7d0bf9e833da5133ea89a8ccc859a297256f3ef4d8d96861fb07e2fb561942";

The problem is that since it's being randomly generated during kernel build time, two kernels with the same derivation hash can end up with two different seeds. Not only is this fundamentaly broken, but that means that all out-of-tree module builds are then "contaminated" by the same non-reproducibility problem: two modules with the same derivation hash ended up being built with two different RANDSTRUCT seeds, causing them to be incompatible with other kernel builds.

I think the only way this can be solved is by having a fixed seed part of the kernel derivation. I'm a complete Nix noob so I'd love to be proven wrong though!

@joachifm

This comment has been minimized.

Copy link
Contributor

commented Jan 8, 2019

What I had in mind is that if the seed is captured when the kernel is initially built, all you should need to do is rig the out-of-tree kernel module build to use that captured seed. It should then not matter that the seed is recreated whenever a build of the kernel is triggered, as you presumably build all of linuxPackages against the same kernel build (and hence the same captured seed). This is quite rigid, however, if you for whatever reason cause the kernel drv to become invalid and so need to be rebuilt, the seed would change and you would be unable to load new modules without rebooting.

@joachifm

This comment has been minimized.

Copy link
Contributor

commented Jan 8, 2019

I had really expected this to Just Work, so there may be a wrinkle in there that I've failed to consider. In any case, I think an externally provided seed is the way to go, even if it should be possible to fix the build process to work the way I've outlined.

@joachifm

This comment has been minimized.

Copy link
Contributor

commented Jan 8, 2019

It also fails obviously if it is not the case that all of linuxPackages was built against the exact same kernel (presumably on the same machine, at the same time). Just a Bad Idea all round, really.

@symphorien

This comment has been minimized.

Copy link
Contributor

commented Jan 9, 2019

The seed could be (deterministically) derived from the store path of the kernel source (or some other kernel related derivation). This fulfills reproducibility and some amount of diversity.

@delroth

This comment has been minimized.

Copy link
Contributor Author

commented Jan 11, 2019

With #53802 merged, RANDSTRUCT is now re-enabled with a seed that's derived from kernel source + config file Nix store path. Let's see if it sticks this time :-)

I'll repurpose this issue for the next step of the plan described earlier: making the seed user-configurable, and adding the required NixOS plumbing to allow users to set their own for their deployments.

@joachifm

This comment has been minimized.

Copy link
Contributor

commented Apr 17, 2019

I've merged @delroth's patch. Thank you.

@joachifm joachifm closed this Apr 17, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.