stdenv: always pass --build, --host to configure

[Gaelan]

Motivation for this change

See #21471 for why this is a good idea. The consensus there seemed to be "go for it," so here's me going for it. This is too big to reasonably test on my laptop, but I imagine this will break some stuff, so it might want to be its own hydra job? I'm not sure how one makes that happen. @edolstra, maybe?

This PR sets configurePlatforms to always default to [ "build" "host" ] in mkDerivation, which was probably the single biggest source of special-casing for cross compiling. However, many individual packages still have special cases; once the dust settles, we should clean those up as well.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[Ericson2314]

Check out https://github.com/NixOS/nixpkgs/pull/44583/files, which was deemed going too far.

[Gaelan]

adding WIP to minimize OfBorg abuse

[Gaelan]

With the perl change, this successfully builds stdenv on macOS.

[xaverdh]

friendly ping

[Gaelan]

Not sure what next steps are here - I'll see if I can still build some basic things with this PR and latest staging, but any serious testing of this requires computing infrastructure I don't have.

[Gaelan]

Ok, rebased against latest staging, successfully build hello (and therefore stdenv) on Darwin; I'm trying the same thing on Linux (in a Docker container) now.

[Ericson2314]

@Gaelan if you can build both stdenvs / hellos, and you have time for the next 2-3 weeks to monitor staging on hydra (something I was never good enough about with my original burst of cross work!!), I think we can merge this. :)

[Gaelan]

Linux hello builds (and runs) successfully as well. I'd be happy to watch Hydra as well, though I'm a little confused about how all that works; there seems to be a hydra job for staging at https://hydra.nixos.org/jobset/nixpkgs/staging, but it looks like it only gets run every few months?

[Ericson2314]

@Gaelan I'm not up to date myself, but I think the idea is that things will be merged into staging-next which is CI'd more.

[Ericson2314]

/rebase staging

[github-actions]

Failed to rebase

[Ericson2314]

In other news, #119625 is working well and merged. That makes me feel like we should finally do this.

I big question is whether passing the same --build and --host makes autoconf think cross or native. If native, if falls short of #21471 but is especially harmless.

[trofi]

To get a bit of incremental progress i'll collect packages known to fail here and possible links to fix them incrementally:

• enchant1:
  • incomplete wrapper shadowing: cc-wrappers should unconditionally provide target prefixed tools ${target}-gcc to shadow unwrapped bianries. #178802
    • cc-wrappers: cc-wrapper: wrap TOOL and ${targetPrefix}-TOOL if both exist #181724
  • invalid OBJC detection in --host= mode.
• radare2: cc-wrapper: wrap TOOL and ${targetPrefix}-TOOL if both exist #181724
• fastjet-contrib: fastjet-contrib: explicitly set 'configurePlatforms = [ ];' #181062
• luarocks: luarocks: explicitly set 'configurePlatforms = [ ];' #178689
• ntl: ntl: explicitly set 'configurePlatforms = [ ];' #181061

[rnhmjoj]

Is anyone still working on this?

[trofi]

I have not seen PRs recently. I think it's still worth doing.

[rnhmjoj]

So, I finally have some free time: what can I do to help get this done?

[trofi]

I would suggest trying configurePlatformsByDefault = true; for your system and try to fix whatever is failing.

radare2 is a good example that shows $tool-gcc (unwrapped) vs gcc (wrapped) mismatch. They have to be the same to not break ./configure. There are a few more issues like that.

[tobim]

I tried to build some stuff on this branch, here is what I got in failures so far:

clisp.log
ctx.log
ekho.log
elf-dissector-unstable.log
etesync-dav.log
grub4dos.log
infamousPlugins.log
keybinder.log
koboredux.log
libftdi.log
netris.log
nokogiri.log
poppler-qt5.log
python310Packages.mlflow-server.log
python310Packages.rfc3986-validator.log
redshift.log
runit.log
shocco.log
shogun.log
tremor.log
tudu.log
yash.log

I planned to start opening PRs over the weekend.
I can share logs is you want them.

[tobim]

I would look for derivations with custom configureScripts as candidates.
Unfortunately we don't know what packages use hand written Makefiles from nixpkgs directly. Those are likely candidates to fail too.

[uri-canva]

@tobim are you trying on this branch, or on master with the commits in this branch rebased on top of it?

[tobim]

I rebased the branch on top of 1460498 (April 5 2023) when I started. I didn't update after that to save time.

[uri-canva]

To prevent regressions would it be helpful to override this at a package level? So we could start building packages in hydra with this set?

[uri-canva]

This fix might be relevant if you're testing changes in cross compiling with this PR: #240435.