[WIP] Rust cross compilation

[Mic92]

Status

• rustc
• fix non-cross rustc
• buildRustPackage
• some packages need fixes & testing

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Fits CONTRIBUTING.md.

---

[Mic92]

Still compiling. Cross-compiled LLVM is also still broken.

[GrahamcOfBorg]

Success on x86_64-linux (full log)

Attempted: libpfm, openssl

Partial log (click to expand)

gzipping man pages under /nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man/share/man/
patching script interpreter paths in /nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man
checking for references to /build/ in /nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/q66rhjlqjkn194c6zji5sjmr3p6z1z6y-openssl-1.0.2p-debug
patching script interpreter paths in /nix/store/q66rhjlqjkn194c6zji5sjmr3p6z1z6y-openssl-1.0.2p-debug
checking for references to /build/ in /nix/store/q66rhjlqjkn194c6zji5sjmr3p6z1z6y-openssl-1.0.2p-debug...
cannot link '/nix/store/.links/0jv8zns4raaa2vc5gjwxqm9n5wx9qi6lbv8556dg6zg80aw3z0yr' to '/nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man/share/man/man3/EVP_PKEY_derive_set_peer.3.gz': No space left on device
cannot link '/nix/store/.links/0jv8zns4raaa2vc5gjwxqm9n5wx9qi6lbv8556dg6zg80aw3z0yr' to '/nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man/share/man/man3/EVP_PKEY_derive_init.3.gz': No space left on device
/nix/store/k4dhk7hqqy3kzmgdvs0j5pi88gfjr4pp-libpfm-4.10.1
/nix/store/pl3km7jksa9r6zrvrwsh5djaxgk004s0-openssl-1.0.2p-bin

[GrahamcOfBorg]

Success on aarch64-linux (full log)

Attempted: libpfm, openssl

Partial log (click to expand)

checking for references to /build/ in /nix/store/818azw7kbva6jznsj438h05lh33r83wv-openssl-1.0.2p...
shrinking RPATHs of ELF executables and libraries in /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man
gzipping man pages under /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man/share/man/
patching script interpreter paths in /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man
checking for references to /build/ in /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/8arzdc863bwrqx50jv441vqfxfbxpjgg-openssl-1.0.2p-debug
patching script interpreter paths in /nix/store/8arzdc863bwrqx50jv441vqfxfbxpjgg-openssl-1.0.2p-debug
checking for references to /build/ in /nix/store/8arzdc863bwrqx50jv441vqfxfbxpjgg-openssl-1.0.2p-debug...
/nix/store/d8g0sv493ar86xvz97bc6ndlv6h3l5g4-libpfm-4.10.1
/nix/store/7y00xv5kwkjvy3pq4npq6finarpyxwc5-openssl-1.0.2p-bin

[matthewbauer]

This is a actually a bug I think. "propagatedNativeBuildInputs" should be correct. Will have a fix soon hopefully.

[Mic92]

I guess I can wait a bit, I just get the things working properly and not blocking on nitpicks.

[Ericson2314]

findXMLCatalogs does sound like a tool that should be propagatedNativeBuildInputs.

[matthewbauer]

I didn't think this would be a problem now that #49608 is merged. Do you know where this comes from? It also looks like openssl supports passing in HASHBANGPERL to do this explicity.

[Mic92]

HASHBANGPERL is only supported in openssl 1.1, but we still use openssl 1.0.2 in most places.

[Mic92]

Also #49608 was reverted again in staging by @LnL7

[LnL7]

Yeah, the stdenv was broken. Since this is the second time I think that change has a bit too much impact to merge into staging directly, we should run it through a separate jobset first to get an overview of the breakages.

[Mic92]

I probably don't need this change to make cross-compiling rust packages work. So I will split it of probably later.

[GrahamcOfBorg]

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: openssl

The following builds were skipped because they don't evaluate on x86_64-darwin: libpfm

Partial log (click to expand)

ld: warning: directory not found for option '-L../bootstrap/x86_64-apple-darwin/usr/lib'
ld: warning: undefined base symbol '__TMC15SwiftFoundation19_NSCFConstantString' for alias '___CFConstantStringClassReference'
installing
post-installation fixup
strip is /nix/store/2ws9cmamvr7xyvdg4d2nnd1bmr1zjrrq-bootstrap-tools/bin/strip
patching script interpreter paths in /nix/store/awcr3pzcd5vpqwh3lhb0ijxbv72vf9l7-swift-corefoundation
cannot build derivation '/nix/store/4xqfqm5lbdqpb9qds1xxrzmyss3818h3-stdenv-darwin.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/mv8zixg5sy9lqnqnn15ca4wl2vxipii6-perl-5.28.0.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/7lipgdn8jvyy7k45idq1sh17k7bpj9z3-openssl-1.0.2p.drv': 2 dependencies couldn't be built
error: build of '/nix/store/7lipgdn8jvyy7k45idq1sh17k7bpj9z3-openssl-1.0.2p.drv' failed

[Mic92]

Mhm, I currently stuck at this fix-point stuff that is used to create buildRustPackage. It somehow puts a non-native rustc/cargo into nativeBuildInputs:

nix-repl> pkgsCross.aarch64-multiplatform.fd.nativeBuildInputs
[ «derivation /nix/store/d9abfxjb0icbjg7k8j62pasgdj7s8zwl-git-2.19.1.drv» «derivation /nix/store/gnvkwv8mls6a2n4bpw3xdpp6fh1d1lyc-cargo-1.30.0-aarch64-unknown-linux-gnu.drv» «derivation /nix/store/hibvax8p73n9imhi9ix7z55ka5hl47ai-rustc-1.30.1-aarch64-unknown-linux-gnu.drv» ]

[Mic92]

Build inputs are clean now.

$ nix-repl> pkgsCross.aarch64-multiplatform.fd.nativeBuildInputs
[ «derivation /nix/store/zp9aqlm1gg9k2hz990yvvwfgja3ybyiy-cargo-1.30.0.drv» «derivation /nix/store/4sjcxw65jn8l90ql2ivn2znlpw24fc09-rustc-1.30.1.drv» «derivation /nix/store/d9abfxjb0icbjg7k8j62pasgdj7s8zwl-git-2.19.1.drv» «derivation /nix/store/8ns0q8dc8bcxyc178ck6zcmhp39rvhwa-nss-cacert-3.40.drv» ]

I think most cross-compile fixes from this pr are actually not needed.
I will separate them later into a new pull request.

[Mic92]

Compilation works now!

$ rustc --target=aarch64-unknown-linux-gnu foo.rs
$ file foo
foo: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/1r1pdrdwgj6k3wy3bhrxjiqjdxf9z2f5-glibc-2.27-aarch64-unknown-linux-gnu/lib/ld-linux-aarch64.so.1, for GNU/Linux 2.6.32, with debug_info, not stripped

[Mic92]

The first working buildRustPackage!

$ file result/bin/fd
result/bin/fd: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/1r1pdrdwgj6k3wy3bhrxjiqjdxf9z2f5-glibc-2.27-aarch64-unknown-linux-gnu/lib/ld-linux-aarch64.so.1, for GNU/Linux 2.6.32, not stripped

[Ericson2314]

Can we do this for the other LLVMs?

[Mic92]

I will split this off into a different pull request: #51226

[Ericson2314]

rustBootstrap should be used as buildPackages.rustBootstrap in (almost?) all cases.

[Ericson2314]

Note these would be buildPackges.cargo buildPackages.rustc, etc, due to splicing black magic.

[Ericson2314]

Oh sorry, rather than changing references to rustBootstrap, makeRustPlatform could use buildPackages.callPackage here. I'd have to stare at the details longer, but the gist of it is all the compilers should come from the previous stage (buildPackages).

[Mic92]

Here too? https://github.com/NixOS/nixpkgs/pull/50835/files#diff-036410e9211b4336186fc613f7200b12R13026

[Mic92]

What is the practical difference of doing so?

[Ericson2314]

So me flipping on the Rust bootstrap stuff aside, here is the actual requirement.

• rustc build != host == target
• buildPackages.rustc build == host != target
• buildPackages.buildPackages.rustc build == host == target

It is tempting to also have:

-buildPackages.buildPackages.buildPackages.rustc prebuilt build == host == target

But sadly we cannot do that as buildPackages^(n+2) = buildPackages^2, as that is they native packages and they are a fixed point.

Solution would be a rustcBinary derivation with a assert that host and target are the same. (Build platform doesn't matter since just unpacking prebuilt.)

The rust and rustBootstrap "platforms" are like stdenv in that they provide something analogous to mkDerivation. Likewise they should use buildPackages.rustc[Binary] just as stdenv uses buildPackage.cc—tools must be sourced from the previous stage.

Also the general pattern in those tools is fooForXXX = XXXPackages.(buildPackages|stdenv).foo, with hostPackages = pkgs. I.e. no need to every be clear about remapping platforms on the level of individual packages, instead use buildPackages and targetPackages to use a package with the *Platforms slid over.