our gcc-unwrapped expression is illegible

[ghost]

@trofi recently blogged (yay rss) about the fact that our gcc expression is... extremely complex. That article gave me more perspective on this comment.

I'd like to start a discussion about trying to get our gcc expression into a state where people can actually understand wtf it is doing. Since we sadly do not have a mailing list anymore, I guess a github issue is the right venue for
long-form discussion.

I'd like to try to keep this focused on simplifying the unwrapped gcc (pkgs/development/compilers/gcc).

Our `cc-wrapper` is extremely complex as well, but there are things it needs to do (like [`NIX_CFLAGS_COMPILE`](https://github.com//issues/79303#issuecomment-720647170)) that look ugly but exist for reasons that took the Nix project a long time to learn. It could use simplification too, but I think the issues there are different and more challenging. However moving things out of `pkgs/development/compilers/gcc` and into the `cc-wrapper` might be part of simplifying `gcc-unwrapped`.

[ghost]

The Compiler

In his blog, @trofi mentions the enormous distance between what we have right now, and the "ideal situation" of ./configure && make && make install. I don't think we can quite make it to that point but we can get very close -- it should be possible to reduce our builder to nothing more than the equivalent of:

env $configureVars ./configure $configureFlags && \
make $makeFlags $makeTargets && \
make install

The lists of flags and variables (configureVars, configureFlags, makeFlags, makeTargets) will always exist, but you can always find out exactly what they are with:

nix show-derivation $(nix-instantiate gcc) | grep configureFlags

Note that you can do the above with zero builds. Those lists should be defined by a simple (but probably quite long) chain of ++ lib.optionals with simple conditions (like targetPlatform.hasSharedLibraries), looking something like:

   # ...
   ] ++ lib.optionals (some-condition) [
     "--with-some-flag"
   ] ++ lib.optionals (another-condition) [
     "--enable-another-flag"
   # ...

Specifically, I think we need to:

1. Stop using a custom builder.sh. I have a PR working towards this.
   • Once this happens the rest of the changes will be much less painful, because right now changing even a single bit in builder.sh triggers a rebuild of stdenv, even if you're just refactoring some condition into an equivalent expression. All this stuff has to go through staging as a result of that.
2. Stop using conditionals in bash code (if..then..fi, case..esac). Conditionals should be lifted into Nix code like the example above.
3. Stop passing environment variables from Nix to the builder, except the standard ones (configureFlags, makeTargets, etc) defined by the standard builder.
4. Stop using nix-support/*-flags to pass things to the gcc expression. This stuff should be passed as Nix variables.

The Target Libraries

@trofi, one thing you called attention to in particular is the way we set -dynamic-linker= in extraLDFlags (which turns into EXTRA_LD_FLAGS_FOR_TARGET). IMHO stuff like this which is Nix-specific probably belongs in the cc-wrapper. Unfortunately the gcc repo is this big monolithic pile of stuff containing both a compiler and a ton of libraries... so in order to set those flags on the target libraries via cc-wrapper we would need to either:

1. Build the target libraries as separate expressions. I started working on this (just for libgcc -- not the other target libraries) but it's extremely invasive.

   • @Ericson2314 told me about Exherbo Linux, which builds some of the target libraries separately, but it's quite high-maintenance... here's what they do for libgcc. Since the gcc build scripts aren't designed for this they have to continually tweak and tune those edo and emake lines with each release. I still don't understand what that magic list of make targets config.h libgcc.mvars tconfig.h tm.h options.h insn-constants.h insn-modes.h gcov-iov.h does. For libgcc this might be worth it since it would break the gcc->glibc->libgcc->gcc circular dependency cycle, but doing it for all of the dozen-ish target libraries... I don't know if we can sustain the maintenance burden there.

2. Trick gcc into using cc-wrapper when compiling its own target libraries. This too is tricky, since we don't build cc-wrapper until after gcc is done building (that could be changed, however -- cc-wrapper could become an additional output of the gcc derivation).

I don't really like either of those options but I can't think of any others. Any ideas?

[trofi]

I think it's important to keep in mind (or even better - document them somewhere) the assumptions, the requirements and the principles of solving complex problems. I see gcc packaging as a complex problem.

I have a few requirements in my head that don't match with current nixpkgs implementation and thus make questions of the implementation details secondary:

1. native bootstrap and cross-bootstrap are quite different today.

   In native case we incrementally build nixpkgs out of bootstrap-tools pieces of toolchain and swap them live. In cross-case we have to cross-compile target gcc and target libc both, and then switch them in a lockstep. I find cross model easier to reason about even if we don't implement it completely today.

2. I think swapping components piecewise is a broken model.

   It just does not work for cross, it produces dubious native results. I find it symptomatic that we have to patchelf ld, wipe references off libunistring, mix binaries from different worlds and play games with sysroot, -B paths to just keep things compiling. And we still get it wrong today.

   Piecewise model makes certain things even more complicated: LTO-bootstrap (built gcc itself with LTO, enable lto plugin in binutils itself) sounds like a simple and natural thing for nixpkgs. But it's hard to implement on top of current scheme: there is always a hazard of mixing new libc from a plugin with old libc from gcc (or the other way around, we never build them both).

   Piecewise model probably breaks our feature detection in gcc as it detects at best wrong (or incomplete) libc's features.

3. Our final result of the bootstrap is still unclean.

   For example we have a runtime closure with 2 libgcc libs:

   $ nix path-info -r $(nix-build -A re2c) |& unnix |& fgrep libgcc
   /<<NIX>>/xgcc-12.3.0-libgcc
   /<<NIX>>/gcc-12.3.0-libgcc
   

   This is a manifestation of gcc->glibc->gcc unbroken dependency circle.

   Or as another example our cross-compilers just don't satisfy basic requirements to even build correctly. For example both pkgsCross.aarch64-multiplatform.gcc11 and pkgsCross.aarch64-multiplatform.gcc13 fail to build against current gcc12 compiler:

   build --no-link -f. pkgsCross.aarch64-multiplatform.gcc11 pkgsCross.aarch64-multiplatform.gcc13 --keep-going
   error: builder for '/nix/store/ma6g1z8r0rjglc5vn5ifsvkmhjad4kdn-gcc-aarch64-unknown-linux-gnu-11.4.0.drv' failed with exit code 2;
       last 10 log lines:
       >     < /build/gcc-11.4.0/libstdc++-v3/../libgcc/gthr-posix.h > aarch64-unknown-linux-gnu/bits/gthr-posix.h
       > sed -e 's/\(UNUSED\)/_GLIBCXX_\1/g' \
       >     -e 's/\(GCC[ABCDEFGHIJKLMNOPQRSTUVWXYZ_]*_H\)/_GLIBCXX_\1/g' \
       >     -e 's/SUPPORTS_WEAK/__GXX_WEAK__/g' \
       >     -e 's/\([ABCDEFGHIJKLMNOPQRSTUVWXYZ_]*USE_WEAK\)/_GLIBCXX_\1/g' \
       >     -e 's,^#include "\(.*\)",#include <bits/\1>,g' \
       >     < /build/gcc-11.4.0/libstdc++-v3/../libgcc/gthr-posix.h > aarch64-unknown-linux-gnu/bits/gthr-default.h
       > make[2]: Leaving directory '/build/build/aarch64-unknown-linux-gnu/libstdc++-v3/include'
       > make[1]: Leaving directory '/build/build'
       > make: *** [Makefile:974: all] Error 2
       For full logs, run 'nix log /nix/store/ma6g1z8r0rjglc5vn5ifsvkmhjad4kdn-gcc-aarch64-unknown-linux-gnu-11.4.0.drv'.
   error: builder for '/nix/store/fqck696srdzk73mlfllflwklaw3ddgwr-gcc-aarch64-unknown-linux-gnu-13.1.0.drv' failed with exit code 2;
       last 10 log lines:
       > make[2]: *** [../../../gcc-13.1.0/libgcc/shared-object.mk:14: truncdfbf2.o] Error 1
       > In file included from ../../../gcc-13.1.0/libgcc/soft-fp/truncsfbf2.c:30:
       > ../../../gcc-13.1.0/libgcc/soft-fp/brain.h:62:1: error: unable to emulate 'BF'
       >    62 | typedef float BFtype __attribute__ ((mode (BF)));
       >       | ^~~~~~~
       > make[2]: *** [../../../gcc-13.1.0/libgcc/shared-object.mk:14: truncsfbf2.o] Error 1
       > make[2]: Leaving directory '/build/build/aarch64-unknown-linux-gnu/libgcc'
       > make[1]: *** [Makefile:14026: all-target-libgcc] Error 2
       > make[1]: Leaving directory '/build/build'
       > make: *** [Makefile:1040: all] Error 2
       For full logs, run 'nix log /nix/store/fqck696srdzk73mlfllflwklaw3ddgwr-gcc-aarch64-unknown-linux-gnu-13.1.0.drv'.
   

   I think this happens because building target-gcc (and it'slibraries) requires the same version of cross-gcc to be present.

With this context I don't think it matters much if you move target libraries out to a separate derivation (and make the build even more deviating from upstream) or use a booter gcc driver even more different from what gcc expects today. Incidentally I removed a standalone libgcc recently because it was completely broken.

Both will probably increase amount of .nix code to "just build the thing" and violate more upstream assumptions along the way. You can violate only so many of them to have a working result :) I don't think solving the problem purely downstream is a viable option.

[Ericson2314]

@trofi So I think once we are finally per-component it really will all be better. I know that sounds like some "after the revolution it will be utopia" sort of thing, but i think the current situation of:

• GCC is not like LLVM
• GCC native is not like GCC cross

really is an awkward middle ground and this is where the pain comes from.

Yes, building GCC per component is not what upstream has in mind, but Exherbo has done it for years (see #132340) and I think it is at least stable. Looking at their code, libgcc is by far the weirdest; the others are pretty simple. Ironically, libgcc is the only we've managed to replicate their way so far.

I have been delaying on #132343 far too long, but I think after 23.11 @amjoseph-nixpkgs and I will pair to try to finish it off --- hold us to it!!

With that done

• GCC will be like LLVM
• GCC native will be like GCC cross (just as LLVM native is like LLVM cross)
• all our component-wise logic will finally start to work much, as the concepts/abstractions will be shared, and we'll finally start to have points of divergence << commonalities

Furthermore, with now two distros going this route, I think we'll have a decent shot at getting upstream to better care about this case. And I fully intended to get any patches etc. submitted upstream around. In particular, no more making the mistake I did of leaving it for later and never getting around to it with the binutils split, I promise!

[ghost]

Hrm, I'm not sure why the merge of #249707 didn't auto-close this.

[trofi]

@Ericson2314 I hope you'll succeed :) I'll refrain from specific comments.