New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] A large batch of cross-compilation fixes #30882

Open
wants to merge 164 commits into
base: staging
from

Conversation

@bgamari
Contributor

bgamari commented Oct 28, 2017

Note: The prefix of commits authored by @Ericson2314 can be ignored for the purposes of this pull request. These are prerequisites for this work but I'll leave upstreaming these to @Ericson2314.

Motivation for this change

Allow Nix to cross-compile a basic system environment.

Things done

The only testing I have done thusfar is to build systemd and its dependencies with the following overlay:

self: super:                                                                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                             
let                                                                                                                                                                                                                                                                                                                          
  pkgs = self.pkgs;                                                                                                                                                                                                                                                                                                          
  callPackage = self.pkgs.callPackage;                                                                                                                                                                                                                                                                                       
in rec {                                                                                                                                                                                                                                                                                                                     
  # guile 2.2 doesn't cross compile. See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28920                                                                                                                                                                                                                                 
  guile = super.guile_2_0;                                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                             
  # We really don't need libapparmor and it is a pain to cross compile due to                                                                                                                                                                                                                                                
  # its perl bindings                                                                                                                                                                                                                                                                                                        
  systemd = super.systemd.override { libapparmor = null; };                                                                                                                                                                                                                                                                  
}                                                                                                 

That being said, this was no small feat. I would like to start getting pieces of this work upstream but I'm a bit uncertain as to how to go about this as there are a few open questions. Namely,

  1. Are we okay with continuing to pepper expressions with doCheck = stdenv.buildPlatform == stdenv.hostPlatform?
  2. Why have I needed to add so many buildPackages. to nativeBuildInputs? Shouldn't this be redundant? Is there perhaps a bug, @Ericson2314?
  3. How do we guarantee that things like fetchurl always use build packages? Is the approach I took here acceptable? It seems something similar will need to be done with unpackCmd.
  4. Is the Always search DT_RPATH hack acceptable?
  5. Some packages' configure scripts have tests that want to run host programs on the build machine. This obviously isn't possible so I provide these test results manually (see, for instance, the krb5 expression). These are of course guesses, but I believe they should apply in most sensible environments. Is this acceptable?

Whatever reviews you can offer are greatly appreciated but I really just wanted to raise a flag so people know this work exists.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@bgamari bgamari requested review from edolstra, FRidh and peti as code owners Oct 28, 2017

@bgamari bgamari referenced this pull request Oct 28, 2017

Merged

Don't run tests when cross-compiling #30883

2 of 8 tasks complete
@bgamari

This comment has been minimized.

Show comment
Hide comment
@bgamari

bgamari Oct 28, 2017

Contributor

I have also opened #30883 which isolates the (hopefully reasonably uncontroversial) doCheck changes.

Contributor

bgamari commented Oct 28, 2017

I have also opened #30883 which isolates the (hopefully reasonably uncontroversial) doCheck changes.

@bgamari bgamari requested a review from zimbatm as a code owner Nov 5, 2017

@zimbatm

This comment has been minimized.

Show comment
Hide comment
@zimbatm

zimbatm Nov 5, 2017

Member

Wow, this is impressive. Is there any way to cut the PR in smaller manageable chunks?

Also, how do I test this? The nixpkgs has some notes on cross-compilation but it's not really clear how to trigger it.

Member

zimbatm commented Nov 5, 2017

Wow, this is impressive. Is there any way to cut the PR in smaller manageable chunks?

Also, how do I test this? The nixpkgs has some notes on cross-compilation but it's not really clear how to trigger it.

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Nov 5, 2017

Member
  1. Once the other PRs contained within here are merged, the changes are fairly self-similar so I don't think breaking up is too important.

  2. I was told at NixCon to provide some easy instantiatations for that.

Member

Ericson2314 commented Nov 5, 2017

  1. Once the other PRs contained within here are merged, the changes are fairly self-similar so I don't think breaking up is too important.

  2. I was told at NixCon to provide some easy instantiatations for that.

@@ -15,7 +15,7 @@ stdenv.mkDerivation rec {
buildInputs = [ libnetfilter_conntrack libnftnl libmnl ];
preConfigure = ''
export NIX_LDFLAGS="$NIX_LDFLAGS -lmnl -lnftnl"
export NIX_LDFLAGS="$NIX_LDFLAGS -lmnl -lnftnl -ldl"
'';

This comment has been minimized.

@dtzWill

dtzWill Nov 5, 2017

Contributor

commit message typo, libld -> libdl

@dtzWill

dtzWill Nov 5, 2017

Contributor

commit message typo, libld -> libdl

This comment has been minimized.

@bgamari

bgamari Nov 5, 2017

Contributor

Thanks! Fixed in my local branch.

@bgamari

bgamari Nov 5, 2017

Contributor

Thanks! Fixed in my local branch.

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Nov 5, 2017

Member

@zimbatm buried within section 5.3 is

nix-build <nixpkgs> --arg crossSystem '(import <nixpkgs/lib>).systems.examples.fooBarBaz

A concrete example would be

nix-instantiate --arg crossSystem '(import ./lib).systems.examples.aarch64-multiplatform' -A systemd

until I do that above that should test this.

Member

Ericson2314 commented Nov 5, 2017

@zimbatm buried within section 5.3 is

nix-build <nixpkgs> --arg crossSystem '(import <nixpkgs/lib>).systems.examples.fooBarBaz

A concrete example would be

nix-instantiate --arg crossSystem '(import ./lib).systems.examples.aarch64-multiplatform' -A systemd

until I do that above that should test this.

@bgamari

This comment has been minimized.

Show comment
Hide comment
@bgamari

bgamari Nov 5, 2017

Contributor

Wow, this is impressive. Is there any way to cut the PR in smaller manageable chunks?

As @Ericson2314 has said, many of the changes are quite similar. That being said, some are not and deserve closer consideration. I can certainly break these out.

Contributor

bgamari commented Nov 5, 2017

Wow, this is impressive. Is there any way to cut the PR in smaller manageable chunks?

As @Ericson2314 has said, many of the changes are quite similar. That being said, some are not and deserve closer consideration. I can certainly break these out.

@zimbatm

This comment has been minimized.

Show comment
Hide comment
@zimbatm

zimbatm Nov 6, 2017

Member

Is there any way to make doCheck = stdenv.buildPlatform == stdenv.hostPlatform automatic? I guess it would require to change the builder.sh. I think that it's too much to ask for normal packagers to remember cross-compilation, we are going to get regressions all the time otherwise.

Member

zimbatm commented Nov 6, 2017

Is there any way to make doCheck = stdenv.buildPlatform == stdenv.hostPlatform automatic? I guess it would require to change the builder.sh. I think that it's too much to ask for normal packagers to remember cross-compilation, we are going to get regressions all the time otherwise.

Show outdated Hide outdated pkgs/stdenv/adapters.nix Outdated
@periklis

This comment has been minimized.

Show comment
Hide comment
@periklis

periklis Nov 6, 2017

Contributor

First of all many many kudos to @bgamari for this batch of insightful work.

@Ericson2314 & @bgamari
I think this PR hits the surface in terms of cross compilation. These changes make visible what work has already been done and what cc patterns for maintainers emerge. We will need a docs/education/migration step for packages to use the cross-compilation pipeline after #29396 and #26805 get merged beyond just explaining all the derivation inputs variables.

I noticed the following patterns for maintainers, please beat me if i misunderstood:

  • Package maintainers have to adapt their doCheck to assert host==build or blacklist tests not cross-compiling
  • Package maintainers have to split their nativeBuildInputs and nativePropagatedBuildInputs because of the minimum dependency for buildPackages.stdenv.cc in the first variable
  • Package maintainers of Autoconf-based packages need to be aware of the autoreconfHook
  • Package maintainers of packages with assembler-generating configure code should be aware of how to manage their build process for cc, e.g. libgcrypt/mpi
  • Package maintainers need to be aware when to use configurePlatforms, e.g. guile
  • Educate maintainers not to hardcode usage of tools from the GNU Toolchain, eg. ar

Also the following hacks are not obvious from a maintenance perspective, but work:

Finally, the following list of packages mentioned in this PR-changeset, should be picked in a separate PR in order to provide a minimum proper cross-compiling base set for package maintainers, in case this PR takes to long to hit master/staging:

P.S. Yes i went through each commit :)

Contributor

periklis commented Nov 6, 2017

First of all many many kudos to @bgamari for this batch of insightful work.

@Ericson2314 & @bgamari
I think this PR hits the surface in terms of cross compilation. These changes make visible what work has already been done and what cc patterns for maintainers emerge. We will need a docs/education/migration step for packages to use the cross-compilation pipeline after #29396 and #26805 get merged beyond just explaining all the derivation inputs variables.

I noticed the following patterns for maintainers, please beat me if i misunderstood:

  • Package maintainers have to adapt their doCheck to assert host==build or blacklist tests not cross-compiling
  • Package maintainers have to split their nativeBuildInputs and nativePropagatedBuildInputs because of the minimum dependency for buildPackages.stdenv.cc in the first variable
  • Package maintainers of Autoconf-based packages need to be aware of the autoreconfHook
  • Package maintainers of packages with assembler-generating configure code should be aware of how to manage their build process for cc, e.g. libgcrypt/mpi
  • Package maintainers need to be aware when to use configurePlatforms, e.g. guile
  • Educate maintainers not to hardcode usage of tools from the GNU Toolchain, eg. ar

Also the following hacks are not obvious from a maintenance perspective, but work:

Finally, the following list of packages mentioned in this PR-changeset, should be picked in a separate PR in order to provide a minimum proper cross-compiling base set for package maintainers, in case this PR takes to long to hit master/staging:

P.S. Yes i went through each commit :)

@zimbatm

This comment has been minimized.

Show comment
Hide comment
@zimbatm

zimbatm Nov 6, 2017

Member

nice review @periklis!

Would it make sense to start with a simple package like hello before touching so many files?

Member

zimbatm commented Nov 6, 2017

nice review @periklis!

Would it make sense to start with a simple package like hello before touching so many files?

@periklis

This comment has been minimized.

Show comment
Hide comment
@periklis

periklis Nov 6, 2017

Contributor

@zimbatm afaik nixpkgs.hello is not representative enough to hit so many important missing parts as @bgamari did with systemd. Imho we need repsentative closures for people that cross-compile by purpose, e.g.

  • the ARM-stuffs needs most parts of a running linux system
  • the Darwin->linux stuffs for docker images needs mostly some libc and build inputs of the final package.
Contributor

periklis commented Nov 6, 2017

@zimbatm afaik nixpkgs.hello is not representative enough to hit so many important missing parts as @bgamari did with systemd. Imho we need repsentative closures for people that cross-compile by purpose, e.g.

  • the ARM-stuffs needs most parts of a running linux system
  • the Darwin->linux stuffs for docker images needs mostly some libc and build inputs of the final package.
@bgamari

This comment has been minimized.

Show comment
Hide comment
@bgamari

bgamari Nov 6, 2017

Contributor

Thanks for the review @periklis! I'll split this up a bit more in the next few days.

Contributor

bgamari commented Nov 6, 2017

Thanks for the review @periklis! I'll split this up a bit more in the next few days.

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Nov 6, 2017

Member

I sort of mentioned this to @bgamari on IRC, but let me reiterate here.

Package maintainers have to split their nativeBuildInputs and nativePropagatedBuildInputs because of the minimum dependency for buildPackages.stdenv.cc in the first variable

  1. The propagated variable is actually propagatedNativeBuildInputs.
  2. buildPackages.stdenv.cc should always be a depsBuildBuild (that wouldn't be propagated).
  3. Currently, there are very few uses of buildPackages.stdenv.cc, just what I and others recently added. They are all nativeBuildInputs as a stop-gap, it is fairly easy to grep and make them depsBuildBuild as I did.
Member

Ericson2314 commented Nov 6, 2017

I sort of mentioned this to @bgamari on IRC, but let me reiterate here.

Package maintainers have to split their nativeBuildInputs and nativePropagatedBuildInputs because of the minimum dependency for buildPackages.stdenv.cc in the first variable

  1. The propagated variable is actually propagatedNativeBuildInputs.
  2. buildPackages.stdenv.cc should always be a depsBuildBuild (that wouldn't be propagated).
  3. Currently, there are very few uses of buildPackages.stdenv.cc, just what I and others recently added. They are all nativeBuildInputs as a stop-gap, it is fairly easy to grep and make them depsBuildBuild as I did.
@periklis

This comment has been minimized.

Show comment
Hide comment
@periklis

periklis Nov 7, 2017

Contributor

@Ericson2314 Thx for re-iterating the information on IRC. What do you think about the rest of the patterns, i identified above? I would be glad to know your opinion on that. Maybe these are too minor to bother.

Contributor

periklis commented Nov 7, 2017

@Ericson2314 Thx for re-iterating the information on IRC. What do you think about the rest of the patterns, i identified above? I would be glad to know your opinion on that. Maybe these are too minor to bother.

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Nov 9, 2017

Member

@periklis sure, I was thinking of going through all of them anywas

  • Package maintainers have to adapt their doCheck to assert host==build or blacklist tests not cross-compiling

#30437 is for this

  • Package maintainers of Autoconf-based packages need to be aware of the autoreconfHook

That hook already exists, to be clear. I suppose its just being added if sufficient support for the host platform we're building for postdates the version of autoconf originally used with the package. In theory this applies to native compilation on that same platform, too?

  • Package maintainers of packages with assembler-generating configure code should be aware of how to manage their build process for cc, e.g. libgcrypt/mpi

I'm not sure what's going on with this, but for code stuff built on the fly in general:

{
  depsBuildBuild = [ buildPackages.stdenv.cc ];
  nativeBuildInputs = [/* libraries needed by code gen tools */];
  buildInputs = [ /* libraries needed at run time */ ];
}
  • Package maintainers need to be aware when to use configurePlatforms, e.g. guile

This should only be compilers, which are rare, hopefully.

  • Educate maintainers not to hardcode usage of tools from the GNU Toolchain, eg. ar

Indeed! haha. And have them don't do LD ?= $(CD)!

A pattern of manual configureFlags/makeFlags wiring seems to emerge in some packages, e.g. gnu-efi: makeFlags and lzip: configureFlags, which seems a brittle approach by not using our wrapper?!?

Yeah. Thankfully, many non-autoconf packages also support these env vars, but yes not all do. I hope that if we always prefix the binaries however, a lack of this stuff will break on cross and native alike, making this impossible to slip through the cracks.

  • Manual -rpath-link usage

I hope those changes are no longer needed with @bgamari's clever patch of ld, partway through here.

Member

Ericson2314 commented Nov 9, 2017

@periklis sure, I was thinking of going through all of them anywas

  • Package maintainers have to adapt their doCheck to assert host==build or blacklist tests not cross-compiling

#30437 is for this

  • Package maintainers of Autoconf-based packages need to be aware of the autoreconfHook

That hook already exists, to be clear. I suppose its just being added if sufficient support for the host platform we're building for postdates the version of autoconf originally used with the package. In theory this applies to native compilation on that same platform, too?

  • Package maintainers of packages with assembler-generating configure code should be aware of how to manage their build process for cc, e.g. libgcrypt/mpi

I'm not sure what's going on with this, but for code stuff built on the fly in general:

{
  depsBuildBuild = [ buildPackages.stdenv.cc ];
  nativeBuildInputs = [/* libraries needed by code gen tools */];
  buildInputs = [ /* libraries needed at run time */ ];
}
  • Package maintainers need to be aware when to use configurePlatforms, e.g. guile

This should only be compilers, which are rare, hopefully.

  • Educate maintainers not to hardcode usage of tools from the GNU Toolchain, eg. ar

Indeed! haha. And have them don't do LD ?= $(CD)!

A pattern of manual configureFlags/makeFlags wiring seems to emerge in some packages, e.g. gnu-efi: makeFlags and lzip: configureFlags, which seems a brittle approach by not using our wrapper?!?

Yeah. Thankfully, many non-autoconf packages also support these env vars, but yes not all do. I hope that if we always prefix the binaries however, a lack of this stuff will break on cross and native alike, making this impossible to slip through the cracks.

  • Manual -rpath-link usage

I hope those changes are no longer needed with @bgamari's clever patch of ld, partway through here.

bgamari and others added some commits Jan 29, 2018

linux/kernel: Use stdenvNoCC for configuration
As described in #34366, the cc-wrapper hook causes stdenv to try creating
$outputDev/nix-support for derivations where $outputDev is a file. This
naturally fails.
XXX: gmp: Disable tests
One test inexplicably fails:

    FAIL t-fac_ui (exit status: 139)
perl-cross: Build with gcc6
It crashes with gcc7.
gcc6: Mark fallthrus explicitly
Otherwise building this fails with -Werror warnings when bootstrapping from
gcc7.
ruby: XXX: Don't use buildRuby
Otherwise gems tries to delete files for the build ruby.
ubootTools: Build envtools as well
This provides fw_printenv and fw_setenv, which are used to manipulate
the bootloader configuration on some platforms.
strongswan: Enable cross-compilation
The ipsec utility provided by strongswan is a shell script. Consequently we need
to patch it to use the target's shell.
nixos: add the strongswan-swanctl service
The strongswan-swanctl systemd service starts charon-systemd. This implements a IKE daemon
very similar to charon, but it's specifically designed for use with systemd. It uses the
systemd libraries for a native integration.

Instead of using starter and an ipsec.conf based configuration, the daemon is directly
managed by systemd and configured with the swanctl configuration backend.

See: https://wiki.strongswan.org/projects/strongswan/wiki/Charon-systemd

Note that the strongswan.conf and swantctl.conf configuration files are automatically
generated based on NixOS options under services.strongswan-swanctl.strongswan and
services.strongswan-swanctl.swanctl respectively.
strongswan-swanctl: support strongswan-5.6.1 configuration options
I determined which options got changed by executing the following
commands in the strongswan repository:

  git diff -U20 5.6.0..5.6.1 src/swanctl/swanctl.opt
  git diff -U20 5.6.0..5.6.1 conf
strongswan-swanctl: don't generate options for charon
This reduces the number of options from 1152 to 756.

@bgamari bgamari referenced this pull request Apr 24, 2018

Merged

A selection of cross-compiling fixes #39420

2 of 8 tasks complete
@dtzWill

This comment has been minimized.

Show comment
Hide comment
@dtzWill

dtzWill May 18, 2018

Contributor

Just curious, are there still pieces of this that need to be "upstreamed"?

Contributor

dtzWill commented May 18, 2018

Just curious, are there still pieces of this that need to be "upstreamed"?

@bgamari

This comment has been minimized.

Show comment
Hide comment
@bgamari

bgamari May 20, 2018

Contributor

There are a few more patches, but they really aren't upstreamable in their current form. Perl still remains the largest issue (#36675). I have reverted the commit cited in that ticket on my branch.

Unfortunately I have been unable to build my rebased branch recently due to an inexplicable stack overflow in the nix interpreter. I'll need to sit down and try to debug this some day.

Contributor

bgamari commented May 20, 2018

There are a few more patches, but they really aren't upstreamable in their current form. Perl still remains the largest issue (#36675). I have reverted the commit cited in that ticket on my branch.

Unfortunately I have been unable to build my rebased branch recently due to an inexplicable stack overflow in the nix interpreter. I'll need to sit down and try to debug this some day.

@Ericson2314 Ericson2314 added this to the 19.03 milestone Aug 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment