Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
RFC: Harden(ed) NixOS #7220
(Spurred by some conversation in #7212, so I decided to write down my thoughts.)
Currently, NixOS and Nixpkgs have a lot of areas in which the security story can improve. While I think we're well positioned to take care of many of these, it's going to take some work, and others we'll need to reach consensus on. (The Good News is that many other people have gone down this road, so we have some good things to learn from)
As a rough starting point, I have identified roughly 5 major areas to look for future enhancements, with my personal suggestions on what we might want to do.
As inspired by #7212, currently, NixOS doesn't default to enabling any hardening capabilities in GCC. There are an array of things we can do here to mitigate exploitability, that we can either enforce as a GCC spec file, or as part of
Suggestion: default all expressions to
Implications: actually, fairly large in theory. These all in conjunction can have a significant penalty to things like startup time (due to resolving dynamic symbols up front), or execution speed. In particular,
Really, only benchmarking these things will tell us. I'd estimate the hit for these doesn't matter for a significant amount of software (common things like
This is issue #2281. I've had this on my queue to finish integrating for a while now; with my new machine this can hopefully be a reality soon enough... Unfortunately as we didn't make the cut for GSoC 2015, there won't be any sponsored work on this. There is still the issue of GCC PGO determinism too, which does not have a clear consensus, but the large majority of the work is elsewhere.
Suggestion: I get off my ass and merge this.
Currently, very few of our NixOS services try to take advantage of any security or isolation features that can be offered e.g. by
Suggestion: We begin enhancing services with these and encouraging maintainers of modules to do the same. Honestly this can probably be done pretty easily and fairly incrementally by maintainers or any interested newcomers.
We should pay attention to upstream systemd units or units from other distributions here too, since they'll have figured out some of this, too.
Kernel security enhancements mostly come from one thing and one thing only (IMO): grsecurity. The good news is that grsecurity support mostly works with my module, and of course it's possible to go out of band with your own custom
One thing is that we don't currently offer prebuilt grsecurity packages. Hydra actually builds them, but there's no way for the module to automagically select the right one. This should be fixed. Futhermore, the binary builds and module need some clean up (e.g.
Also, my grsecurity module could use a bit of work. Unfortunately it's split up in several places due to needing build/module support, but I do think this could be cleaned up/refactored a bit.
There are other things to consider, too. For example, there are kernel patches floating around to do various other things; it would be really nice, for example, if we could have a patch to randomize the MAC address assigned by the kernel - this would be good for my laptop and could be controlled by a boot switch for the kernel.
Suggestion: Well, it mostly works I guess.
NixOS currently doesn't offer any form of policy enforcement in the place of MAC systems. There are a lot to choose from, but it basically comes down to AppArmor vs SELinux. Right now there's nobody really supporting either, so in lieu of this, the support that does exist is geared towards AppArmor. AppArmor isn't as expressive as SELinux, but it's a hell of a lot simpler and the policies are far easier to maintain. I think this is pretty important to get people to write policies.
The good news is the actual infrastructure is there: apparmor is packaged and works. There are even NixOS modules for it, but nothing really uses it. This is pretty easy though: we can begin clipping apparmor policies from upstream packages and places like Ubuntu.
Suggestion: We should just start writing policies, and enforce AppArmor by default on NixOS. This is the best way to ensure people keep using it, IMO.
From previous discussions, some of the flags (with negligible performance disadvantages) should be easily accepted to be in the defaults, perhaps even without an explicit option to disable them. Some regular distros also do have some of these for all packages IIRC.
Side note: I've seen that generating non-deterministic binaries is also used to improve security. I've seen that elsewhere, but wiki also mentions randomization of control flow https://en.wikipedia.org/wiki/Binary_hardening. (I'm no security guy, but I expect such techniques aren't available with unmodified toolchain, and non-determinism is a bit difficult with nix anyway.)
@thoughtpolice in addition to all your excellent points, I've also wondered about reliable "auditability" of the system. The recent work on #7092 helps figure out what
@vcunat I think 99% of the security benefits you'd get from generating nondeterministic binaries are attained by ASLR on deterministic binaries. That is, unless your threat model involves an attacker using tooling that assumes binaries are laid out in a specific manner (and doesn't otherwise interpret ELF or your object format), what matters is runtime memory layout, not static binary layout. That's not to say that other aspects of binary hardening aren't applicable. If we have no source for e.g., Spotify, there might be instances in which the binary can be modified and made more secure, but that seems like a much less common case for a distro that is almost 100% source-based.
FWIW, I am in general agreement with @thoughtpolice's suggestions.
I would add that it should be a no-brainer to allow the implementation of the security enhancements that have already been done by any of the mainstream, general-purpose distros that rely on a large community of contributors (e.g. Debian, Ubuntu, Arch Linux, non-hardened Gentoo, ...). The last time I looked, Ubuntu in particular implemented a significant number of these enhancements. These have already been battle-tested and shouldn't have a significant impact in maintaining most packages.
Also, I think it would be good to prioritize the enhancements @arno01 has been working on in #7212 and related PRs (although, I guess we should try applying them to most of NixOS instead of on a package-by-package basis, assuming that's what we want). This is obviously a priority to him and it can be frustrating and discouraging if we don't accept his contributions (or at least, help / guide him) for a long time. The same applies to any other already-implemented contributions / contributors.
I assume this has already been thought of, but: if there will be that kind of penalty, we need to allow per-package opt-out. We package numerical computing libraries for which that kind of penalty is unacceptable, whatever the security implications. The same thing goes for PIE. I'm not asking for an unhardened system, just some variable for
Some thoughts: Yes, I fully envision this will be controllable via a flag for
The granularity of these options is up for debate; for example RELRO/BIND_NOW protection can probably always be enforced.
Disabling PIE and stack protection will be the biggest wins for most applications. Note that PIE is very efficient on x86_64 in comparison to i386; in fact, stack protection will be the biggest loss on 64bit, and PIE should have little effect. It's only 32bit where they both hurt.
Numerical notes: note that PIE is only going to hurt applications - this is a very important distinction for numerical libraries, because PIE is basically just a dynamically shared object disguised as ELF. Meaning, if you already had a numerical library on 32bit machines that offered a
However, it's not clear to me how high the demand for 32bit numerical libraries are anyway in this scenario anyway - almost anyone seriously using them is going to opt for a 64bit instruction set which will offer wider register sets and more features. Finally, things like dynamically linked copies of
I think where this is really going to be a bigger impact is things like databases, which aren't hand written assembly, but a very large performance sensitive code base.
I suppose for this we'll have to extend the Hardened GCC patches if we decide to go that route, because then
Thanks for the feedback, let me know if you think this sounds all wrong. :)
I think we need something like smart-flexi-hardening. It should be relatively easy to enable hardening global-wise or per-package-wise including the levels of hardening, for example:
On Tuesday, April 07, 2015 14:10:20 Austin Seipp wrote:
Binary hardening: this doesn't sound desirable if the performance impact is that bad. Could be an option for network-facing services.
Kernel enhancements: I'm reluctant to depend too much on non-standard kernel patches, since they can create a situation where we can't update our kernel to the latest upstream version because the patches no longer apply. E.g. the discussion at #6858 seems to imply that grsecurity does not support 3.18 kernels, meaning we can't use that (LTS) branch.
Enabling AppArmor by default: Sounds good.
The problem is that this does not scale nicely for users - they really want binary packages, and every single one of these different flags implies a complete rebuild of pretty much the entire transitive closure of your system environment, because it will imply hash changes all the way through the graph your system closure. Selecting appropriate defaults for users is extremely important so they get A) good improvements with B) minimal friction and pain. The current default is essentially equivalent to your L0, which I find unoptimal. :)
In general I'm OK with leaving judgement to maintainers of packages/services about what hardening steps are necessary, but I'm much more hesitant about leaving that decision up to users, because the trade offs are much less clear than they may seem.
The impact is really only PIE/stack protection. The others are very cheap and increase security with few - if any - complications. Again, I think the correct thing to do is to have services that absolutely need the extra performance opt-out of these two features. Just looking at my system closure for example, there are a lot of applications for which this would only increase security with few downsides - systemd, coreutils, tcpdump/libpcap, etc etc. In fact, I'd say network services are the ones that want to opt out, not in, precisely because they're performance sensitive and tend to have a large amount of security scrutiny anyway.
I also like to keep in mind almost every other distribution enforces many of these enhancements anyway. This performance impact may seem bad, but it's a loss that a very high majority of distribution maintainers see as appropriate. In fact, PIE is the only truly controversial one.
An alternative would be to instead default on
PIE is something special on its own, and perhaps worth not enabling by default on 32bit (which is a bit weird and inconsistent), but on 64bit, should absolutely be enforced IMO as it can significantly complicate code-reuse exploits and RIP-relative addressing is cheap in comparison to stealing a base register.
Just to be clear: I do not ever plan on making NixOS require things like grsecurity* - which, on its own, would be a bigger discussion than this IMO - which is an extensive and massive patchset. (Something like a MAC randomization patch is, OTOH, about 20 lines of code), and it also slightly complicates the LTS story as I indicated in that other thread. The support we have now would need extensive polishing in any case.
We all know that buffer overflows (not only) are common in the C and C++ languages and every qualified security expert knows that each and every protection mechanism -- matters. Having just one single weak point can lead to a system compromise.
RBAC can be implemented in AppArmor via apparmor-pam, which would benefit everybody not just grsec users. AppArmor also has what looks to be a fairly comprehensive learning mode (all of that is currently broken on NixOS, though). All-in-all, it seems most cost-effective to focus on AppArmor for RBAC and automatic learning and rely on grsec mostly for PaX et al.
As an update, I have a branch (not yet published) which:
Here's an example of an "Improved" grsecurity description using NixOps (NB: not representative of final overhaul):
and still get proper service, udev rules, etc.
I still need to do some extra stuff. In particular this current interface doesn't offer any help in configuring kernel parameters or setting configurations, so it's definitely not merge ready. Also, I have only just begun adding AppArmor and systemd policies, so I'll probably bundle/test more of them, too.
Please let me know what you think. I'll open a PR later this week.
Speaking on security "out-of-a-box", right now I am building a standard (non-hardened) LFS where I notice that systemd 219 by default enables all common protections like Canary (SSP with the buffer-size=4), PIE, Full RELRO and FORTIFY_SOURCE:
[ http://www.linuxfromscratch.org/lfs/view/stable-systemd/chapter06/systemd.html ]
I'm removing the milestone here as the description of the PR is fairly outdated. Is there anyone willing to take this meta-issue forward?
It's important to have an overview, but it doesn't help if it's outdated information since a year, that will mislead bycomers.
If there's noone willing to clean this up, I'll just close it in a few weeks.
Hello. I have updated to NixOS 16.09 and I have difficulty using GCC in it.
GCC inserts stack protector checks into my program. I can not disable it (there is no option -fno-stack-protector-strong provided by GCC).
I don't mind if packages will be built with hardening by default. But is it a bit inconvenient that the default gcc command enables hardening with no clear way to disable it.