GCC has unwanted flags

[andrewrk]

andy@xps:~/tmp$ NIX_DEBUG=true gcc
(some good flags for dealing with /nix/store/* and then...)
  -O2
  -D_FORTIFY_SOURCE=2
  -fstack-protector-strong
  --param
  ssp-buffer-size=4
  -fPIC
  -fno-strict-overflow
  -Wformat
  -Wformat-security
  -Werror=format-security

These flags should be enabled when compiling nix packages, but they should not be always on for general compiler use.

This has caused a broken debugging experience for me on the project I was working on.

Most projects have a "debug build" that has optimizations off, and this prevents that from working correctly.

NixOS version 16.09pre90254.6b20d5b (Flounder)

[andrewrk]

I believe this was caused by #12895

[andrewrk]

Not only that, but these flags are in the extraAfter section, which means I can't even override the flags, for example with -Og in a Makefile or build script.

[andrewrk]

One workaround is putting

    hardeningDisable = [ "fortify" ];

in a nix-shell. Since I was already using a nix-shell for my environment, this is an acceptable workaround. Still, seems weird to assert -O2 by default and have to disable it to get previously expected behavior.

[abbradar]

I think this can be fixed by adding a special flag, NIX_ENFORCE_HARDENING, that is only set in stdenvs. This way the flags won't be enforced in regular gcc invocations and could be disabled in shells by setting this flag to 0.

BTW having -O2 always added is not nice because some applications might want -O3 or -Ofast (which would be overridden). We may want to make this conditional somehow...

[rasendubi]

/cc @globin @fpletz

[fpletz]

You can use

hardeningDisable = [ "all" ];

to disable all hardening flags. This should also work if set as an environment variable.

We will work on a solution after 16.09 using gcc spec files that can detect for example if libraries are built or debugging is enabled. This was just our first iteration and we agree that it is not perfect. Changes to the cc-wrapper always require a full rebuild, which is very painful. More feedback on #12895 would've been helpful.

We didn't anticipate and test that those flags would be propagated to regular gcc invocations outside of nix builds. Not entirely sure how to fix it yet but @abbradar's proposal sounds reasonable. I will look into it after 16.09.

-O2 is required for -D_FORTIFY_SOURCE=2. Not sure though if higher optimizations also work.

[abbradar]

FWIW I found that level 1 fortifying requires inlining:

As I said in that post, the special fortified functions (those that are available in the form of __$func_chk in the libc.so file and provide warnings at build time, and proper stack traces at runtime) only get enabled if inline functions are enabled, so are totally ignored at -O0 (simply disabling inlines, by using -fno-inline won’t stop them from being used, though).

I haven't been able to find anything except of anecdotal evidence that -O2 is required for level 2 (I'm not questioning your choice, rather I was trying to find a list of optimizations that are required so that we can enable them individually).

[fpletz]

(I'm not questioning your choice, rather I was trying to find a list of optimizations that are required so that we can enable them individually).

I'm always open for suggestions. Thanks! 😃 We'll have to try that.

[abbradar]

Unfortunately it may need peeking into rat nest GCC/glibc's codebases to determine this. Meanwhile I think it's okay to leave -O2.

[copumpkin]

Just as a quick reminder when you talk about gcc: there's also clang to consider on Darwin (and in some cases on Linux too)

[jxy]

So this is why recently gdb suddenly tells me different things in my code and gcc no longer does a good job optimizing my code. I almost thought gcc 5.4 had optimization regressions.

How do I get back a reasonable working gcc? Which nixpkgs version should I revert to?

Does this mean I should put hardeningDisable = [ "all" ]; in all of my custom nix files for my HPC production code, eg, mpich3?

I'm using nixpkgs on a ubuntu without root privilege.

It is NOT okay to leave -Oanything in extraAfter!

[andrewrk]

We're all in agreement here @jxy and going to fix it soon. This is why it's called unstable!

The workaround you mentioned is working for me and is probably the reasonable thing to do until this is fixed.

[jxy]

Right, thanks for the workaround. It wasn't easy to find this thread. I guess I'll need some performance regression tests after changing my system.

[jxy]

Does NixOS 16.09 release version have this issue? If so, I'll put off upgrading until this is fixed.

[vcunat]

Yes, both 16.09 and unstable/master.

[fpletz]

I'm currently working on a fix.

[andrewrk]

even though I knew about this, I temporarily forgot and it caused me to file a bogus issue report on another project: thejoshwolfe/legend-of-swarkland#36

[mboisson]

I'm running into this issue when compiling older versions of GCC.

gcc 4.8 compiles fine, but at run time, it tries to use stack-protector-strong, which is not supported.

I've wasted about 8 hours so far trying to fix this broken thing....
[mboisson@build-node easybuild-easyconfigs]$ gcc --version
gcc: erreur: unrecognized command line option ‘-fstack-protector-strong’
gcc (GCC) 4.8.5

Our fork of nixpkgs is based on 16.09 and is here:
github.com/computecanada/nixpkgs

is there any commit I could pull to fix this ?

[globin]

There is no optimal fix yet, workaround is still to export hardeningDisable=all in your shell while developing.

[mboisson]

Thanks.

[YellowOnion]

Trying to code with -Werror is stupid and annoying, how the hell do you disable it?

[RaunakChhatwal]

Why is nobody mentioning the fact that it is completely and totally unacceptable that these flags are passed when the stated purpose of the wrapper is to pass include and link paths? I just lost a whole day to trying to figure out why clang is acting up! Why can't compilers work like in any other distro, with the wrapper only passing include and link nix store paths?

[vcunat]

And other flags would be added by yet another layer of wrapping?

[RaunakChhatwal]

How about create a package called llvmPackages_16.clang_hardened with the wrapper that is currently in clang?

[vcunat]

I don't know... manually adding yet another attribute for each compiler doesn't sound attractive to me, but something similar might perhaps be done.

[zopsicle]

There is already gcc-unwrapped and clang-unwrapped, which do not have a wrapper. They only suffer from not being able to find headers and libraries. If there were a derivation that outputs a sysroot that could be passed to the compiler with --sysroot, that includes crucial stuff like libc, crt0, etc., that would pretty much solve the problems I think. (Maybe this is an oversimplification. There's also -B, not really sure what that does.)

cc-wrapper also forwards $NIX_CFLAGS_COMPILE to the compiler. This variable is set by stdenv's builder and contains -I flags and the like. Depending on your use case, this may or may not be useful, but nonetheless it can easily be passed explicitly if desired. (I personally never want this because I use CMake or pkg-config to find libraries.)

[RaunakChhatwal]

I most lean towards a solution where compilers work in nixos out of the box like they would in any other distro, which probably will still require a wrapper passing header and link nix store paths since nix profiles don't have an include directory. Maybe just change hardeningDisable to be all by default?

[andrewrk]

why can't you just pass the flags when building nix packages but not pass them otherwise? seems pretty straightforward to me.

[RaunakChhatwal]

Exactly, and it seems to me that the least disruptive way to do this would be to set the default hardeningDisable to [ "all" ] and all nix packages explicitly set hardeningDisable = [ ];

[vcunat]

No, I don't see it as simply. It's mostly described above in years old posts, but let me reiterate shortly. When you build a package with nixpkgs stdenv (via nix-shell, nix-build, etc.), naturally everything is as close as possible to building an actual nixpkgs package. That includes the hardening options that are default in nixpkgs packages.

First we'd have to find a good way how to tell if the user's intention is to contribute to nixpkgs (i.e. use nixpkgs hardening) or something else. And even then I'd think hard whether it's worth creating another set of default behavior.

(I'm not really considering running a compiler outside of nix-shell or nix-build. It isn't well usable when you need libraries.)

[mboisson]

We tried for years to use Nix as a basis for something else on top, and because of issues like this, we just gave up. Nix is all beautifully coherent when you build within Nix, not when you want to build on top of Nix. My recommendation is if you want to use Nix, use it all the way, don't build things outside of it. If you don't want to do that, use something else than Nix (we went for Gentoo Prefix).

[SohamG]

Exactly, and it seems to me that the least disruptive way to do this would be to set the default hardeningDisable to [ "all" ] and all nix packages explicitly set hardeningDisable = [ ];

I would like to point out that this is simply not enough. I came across an issue when compiling the toy OS for my OS Class where the result of standalone compiled ELF binary was larger than the max file size supported in the OS. The instructor and other classmates (all on non-nix systems) did NOT have this issue. I can reproduce this with the nix-shell settings quoted.

[RaunakChhatwal]

No, I don't see it as simply. It's mostly described above in years old posts, but let me reiterate shortly. When you build a package with nixpkgs stdenv (via nix-shell, nix-build, etc.), naturally everything is as close as possible to building an actual nixpkgs package. That includes the hardening options that are default in nixpkgs packages.

First we'd have to find a good way how to tell if the user's intention is to contribute to nixpkgs (i.e. use nixpkgs hardening) or something else. And even then I'd think hard whether it's worth creating another set of default behavior.

(I'm not really considering running a compiler outside of nix-shell or nix-build. It isn't well usable when you need libraries.)

Given that nix-shell is touted as a development environment, it comes as a surprise to me that everything has to be as close as possible to production. As you mentioned, compilers outside of nix-shell are unusable due to the missing headers, so development needs to happen in nix-shell.

[andrewrk]

I think hardening should be deleted entirely. It's too aggressive. You shouldn't just apply random flags to applications that the upstream authors didn't ask for. If they wanted hardening flags they would have put them upstream, too.

It's also a half-measure; a false sense of security, just like ASLR. We should move on to better things.

[vcunat]

Given that nix-shell is touted as a development environment, ...

nix-shell was designed to provide as close environment to nix-build as practical for development.

[Artturin]

Vars for only nixpkgs packages (no nix-shell) can be set at genericBuild
Currently, it's done for no gzip timestamps

nixpkgs/pkgs/stdenv/generic/setup.sh

Lines 1543 to 1545 in 680602e

	# variable used by our gzip wrapper to add -n.
	# gzip is in common-path.nix and is added to nix-shell but we only want to change its behaviour in nix builds. do not move to a setupHook in gzip.
	export GZIP_NO_TIMESTAMPS=1

[WesleyAC]

Just ran into this problem, and I agree that this hardening code should be deleted, or at the very least turned off be default. Overriding upstream compiler flags by default for everything is not a good idea.

[chrispickard]

just lost two days to this bug, I'm really not sure why we are configuring gcc differently than everyone else outside of just ensuring it doesn't load from "system" directories. the error messages I got weren't even close to the issue (which is that fortify was turned on) so it took a long time to search random phrases and then eventually stumble on this issue. we have to do something about this. I believe I have even run into this bug before I just didn't realize it at the time.

[SohamG]

Chris Pickard ***@***.***> writes:

I got weren't even close to the issue (which is that fortify was turned on) so it took a long time to search random phrases and then eventually stumble on this issue. we have to do something about this.

Is there a problem of this issue not being on the radar of people "in charge" of gcc in nixpkgs? Can't imagine that to be the case, yet all I see is this thread getting longer with more frustration and no update from the toolchain derivation maintainers. To be fair this issue was incorrectly marked as resolved a while ago...

[bjornfor]

AFAIR, nobody had a good solution to this issue until october this year: #18995 (comment). Someone "just" have to take that idea and apply it to the compiler wrapper?

[SohamG]

Bjørn Forsman ***@***.***> writes:

AFAIR, nobody had a good solution to this issue until october this year: #18995 (comment). Someone "just" have to take that idea and apply it to the compiler wrapper?

I just took a look - I think this issue is to do with the flags "baked in" to GCC when its built and therefore cannot be overriden with environment variables, especially cause the program being built would set $CFLAGS etc for itself. I think the solution is a different GCC derivation for the builder environment, and a setting in nix shell to choose a "normal" GCC or the one with the extra flags.

[jmarmstrong1207]

Any updates @fpletz @cstrahan ?

[jmarmstrong1207]

I'm not sure what the issue is though. These flags can be disabled and they are enabled by default not just in NixOS, but also in other distros such as Fedora and Ubuntu to increase security. Is there something I'm missing?

[zopsicle]

Other distroes enable these flags when building packages, but not for general purpose compiler use (i.e. if the user is a programmer writing their own programs, not distro packages). Nixpkgs wraps the compilers to always set these flags, regardless of use case.

[vcunat]

Yes, but Nix generally do not split these use cases. The normal way to do development is that you get nix-shell with everything set up as if doing a nix package build.

[zopsicle]

I see. Then, is there a tool that can provide development environments with Nix packages, for when you’re not building a Nix package? I think "developing software that will exist outside the Nix ecosystem, using tools provided by Nix in an ephemeral environment" is a use case that should be officially supported, and is in fact something a lot of people currently abuse (apparently) nix-shell for.

[jmarmstrong1207]

I believe distrobox is a pretty good alternative and is what you're looking for