[WIP] Rust cross compilation #50866

Mic92 · 2018-11-21T01:58:21Z

Mic92 · 2018-11-21T01:59:01Z

Still compiling. Cross-compiled LLVM is also still broken.

GrahamcOfBorg · 2018-11-21T02:17:03Z

Success on x86_64-linux (full log)

Attempted: libpfm, openssl

Partial log (click to expand)

gzipping man pages under /nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man/share/man/
patching script interpreter paths in /nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man
checking for references to /build/ in /nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/q66rhjlqjkn194c6zji5sjmr3p6z1z6y-openssl-1.0.2p-debug
patching script interpreter paths in /nix/store/q66rhjlqjkn194c6zji5sjmr3p6z1z6y-openssl-1.0.2p-debug
checking for references to /build/ in /nix/store/q66rhjlqjkn194c6zji5sjmr3p6z1z6y-openssl-1.0.2p-debug...
cannot link '/nix/store/.links/0jv8zns4raaa2vc5gjwxqm9n5wx9qi6lbv8556dg6zg80aw3z0yr' to '/nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man/share/man/man3/EVP_PKEY_derive_set_peer.3.gz': No space left on device
cannot link '/nix/store/.links/0jv8zns4raaa2vc5gjwxqm9n5wx9qi6lbv8556dg6zg80aw3z0yr' to '/nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man/share/man/man3/EVP_PKEY_derive_init.3.gz': No space left on device
/nix/store/k4dhk7hqqy3kzmgdvs0j5pi88gfjr4pp-libpfm-4.10.1
/nix/store/pl3km7jksa9r6zrvrwsh5djaxgk004s0-openssl-1.0.2p-bin

GrahamcOfBorg · 2018-11-21T02:19:06Z

Success on aarch64-linux (full log)

Attempted: libpfm, openssl

Partial log (click to expand)

checking for references to /build/ in /nix/store/818azw7kbva6jznsj438h05lh33r83wv-openssl-1.0.2p...
shrinking RPATHs of ELF executables and libraries in /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man
gzipping man pages under /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man/share/man/
patching script interpreter paths in /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man
checking for references to /build/ in /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/8arzdc863bwrqx50jv441vqfxfbxpjgg-openssl-1.0.2p-debug
patching script interpreter paths in /nix/store/8arzdc863bwrqx50jv441vqfxfbxpjgg-openssl-1.0.2p-debug
checking for references to /build/ in /nix/store/8arzdc863bwrqx50jv441vqfxfbxpjgg-openssl-1.0.2p-debug...
/nix/store/d8g0sv493ar86xvz97bc6ndlv6h3l5g4-libpfm-4.10.1
/nix/store/7y00xv5kwkjvy3pq4npq6finarpyxwc5-openssl-1.0.2p-bin

matthewbauer · 2018-11-21T02:43:35Z

Sign in to view

-  nativeBuildInputs = [unzip];
-  propagatedNativeBuildInputs = [ findXMLCatalogs ];
+  nativeBuildInputs = [ unzip ];
+  propagatedBuildInputs = [ findXMLCatalogs ];


This is a actually a bug I think. "propagatedNativeBuildInputs" should be correct. Will have a fix soon hopefully.

I guess I can wait a bit, I just get the things working properly and not blocking on nitpicks.

findXMLCatalogs does sound like a tool that should be propagatedNativeBuildInputs.

matthewbauer · 2018-11-21T02:55:17Z

Sign in to view

@@ -105,6 +105,10 @@ let
    '';

    postFixup = ''
+      # fixes cross-compilation to pick the right perl,
+      # since perl is also in nativeBuildInputs
+      (sed -i -e '1 s!^.*$!${perl}/bin/perl!' $bin/bin/c_rehash)


I didn't think this would be a problem now that #49608 is merged. Do you know where this comes from? It also looks like openssl supports passing in HASHBANGPERL to do this explicity.

HASHBANGPERL is only supported in openssl 1.1, but we still use openssl 1.0.2 in most places.

Also #49608 was reverted again in staging by @LnL7

Yeah, the stdenv was broken. Since this is the second time I think that change has a bit too much impact to merge into staging directly, we should run it through a separate jobset first to get an overview of the breakages.

I probably don't need this change to make cross-compiling rust packages work. So I will split it of probably later.

GrahamcOfBorg · 2018-11-21T03:18:10Z

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: openssl

The following builds were skipped because they don't evaluate on x86_64-darwin: libpfm

Partial log (click to expand)

ld: warning: directory not found for option '-L../bootstrap/x86_64-apple-darwin/usr/lib'
ld: warning: undefined base symbol '__TMC15SwiftFoundation19_NSCFConstantString' for alias '___CFConstantStringClassReference'
installing
post-installation fixup
strip is /nix/store/2ws9cmamvr7xyvdg4d2nnd1bmr1zjrrq-bootstrap-tools/bin/strip
patching script interpreter paths in /nix/store/awcr3pzcd5vpqwh3lhb0ijxbv72vf9l7-swift-corefoundation
cannot build derivation '/nix/store/4xqfqm5lbdqpb9qds1xxrzmyss3818h3-stdenv-darwin.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/mv8zixg5sy9lqnqnn15ca4wl2vxipii6-perl-5.28.0.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/7lipgdn8jvyy7k45idq1sh17k7bpj9z3-openssl-1.0.2p.drv': 2 dependencies couldn't be built
error: build of '/nix/store/7lipgdn8jvyy7k45idq1sh17k7bpj9z3-openssl-1.0.2p.drv' failed

Mic92 · 2018-11-21T12:00:18Z

Mhm, I currently stuck at this fix-point stuff that is used to create buildRustPackage. It somehow puts a non-native rustc/cargo into nativeBuildInputs:

nix-repl> pkgsCross.aarch64-multiplatform.fd.nativeBuildInputs
[ «derivation /nix/store/d9abfxjb0icbjg7k8j62pasgdj7s8zwl-git-2.19.1.drv» «derivation /nix/store/gnvkwv8mls6a2n4bpw3xdpp6fh1d1lyc-cargo-1.30.0-aarch64-unknown-linux-gnu.drv» «derivation /nix/store/hibvax8p73n9imhi9ix7z55ka5hl47ai-rustc-1.30.1-aarch64-unknown-linux-gnu.drv» ]

Mic92 · 2018-11-21T14:11:40Z

Build inputs are clean now.

$ nix-repl> pkgsCross.aarch64-multiplatform.fd.nativeBuildInputs
[ «derivation /nix/store/zp9aqlm1gg9k2hz990yvvwfgja3ybyiy-cargo-1.30.0.drv» «derivation /nix/store/4sjcxw65jn8l90ql2ivn2znlpw24fc09-rustc-1.30.1.drv» «derivation /nix/store/d9abfxjb0icbjg7k8j62pasgdj7s8zwl-git-2.19.1.drv» «derivation /nix/store/8ns0q8dc8bcxyc178ck6zcmhp39rvhwa-nss-cacert-3.40.drv» ]

I think most cross-compile fixes from this pr are actually not needed.
I will separate them later into a new pull request.

Mic92 · 2018-11-25T21:46:36Z

Compilation works now!

$ rustc --target=aarch64-unknown-linux-gnu foo.rs
$ file foo
foo: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/1r1pdrdwgj6k3wy3bhrxjiqjdxf9z2f5-glibc-2.27-aarch64-unknown-linux-gnu/lib/ld-linux-aarch64.so.1, for GNU/Linux 2.6.32, with debug_info, not stripped

Mic92 · 2018-11-28T09:20:11Z

The first working buildRustPackage!

$ file result/bin/fd
result/bin/fd: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/1r1pdrdwgj6k3wy3bhrxjiqjdxf9z2f5-glibc-2.27-aarch64-unknown-linux-gnu/lib/ld-linux-aarch64.so.1, for GNU/Linux 2.6.32, not stripped

Ericson2314 · 2018-11-28T23:28:07Z

Sign in to view

-    "-DTARGET_TRIPLE=${stdenv.hostPlatform.config}"
-
+    "-DLLVM_DEFAULT_TARGET_TRIPLE=${stdenv.targetPlatform.config}"
+    "-DLLVM_TARGETS_TO_BUILD=${llvmTarget stdenv.hostPlatform};${llvmTarget stdenv.targetPlatform}"


Can we do this for the other LLVMs?

I will split this off into a different pull request: #51226

Mic92 · 2018-12-18T12:46:25Z

pkgs/development/compilers/rust/rustc.nix

+
+    "${setBuild}.jemalloc=${hostJemalloc}/lib/libjemalloc_pic.a"
+    "${setHost}.jemalloc=${hostJemalloc}/lib/libjemalloc_pic.a"
+    "${setTarget}.jemalloc=${targetJemalloc}/lib/libjemalloc_pic.a"


@Ericson2314 this one seems actually undefined, when I try to build nix-build default.nix -A pkgsCross.aarch64-multiplatform.rustc

Yeah that's OK. For now. I need to change something in booter.nix to avoid these issues. Need to throw in a buildPackages to get the cross compiler vs cross compiler compiler, which is the immediate need anyways.

Ericson2314 · 2019-01-02T17:13:47Z

Is this just blocked on ofborg?

Mic92 · 2019-01-02T19:01:31Z

It also requires to fix a few cargo packages that depend on ./target/release to be present, as now cross-compiled packages are put into a architecture subdirectory that can be accessed by using $releaseDir

Ericson2314 · 2019-01-02T19:10:42Z

If you want to list those, I'm happy to divide + conquer on fixing that + eval failures.

Mic92 · 2019-01-02T19:50:19Z

I will not have a look at this the next few days, so you could just fix the
few packages right away as it is only string-replacement in 11 packages. (target/release -> $releaseDir)

$ ag -l 'target/release'
pkgs/development/tools/rust/racerd/default.nix
pkgs/development/tools/misc/ycmd/default.nix
pkgs/build-support/rust/default.nix
pkgs/tools/text/ripgrep/default.nix
pkgs/tools/misc/vdirsyncer/default.nix
pkgs/tools/misc/fd/default.nix
pkgs/tools/misc/bat/default.nix
pkgs/applications/altcoins/zcash/librustzcash/default.nix
pkgs/applications/networking/dyndns/cfdyndns/default.nix
pkgs/applications/misc/alacritty/default.nix
pkgs/applications/window-managers/wtftw/default.nix

This should probably also mentioned in the changelog since this also affects out-of-tree rust packages.

Ericson2314 · 2019-01-18T15:44:20Z

We talked briefly about this on IRC. I think it should action be changed back to target/release since we only install for a single platform at once. This could be done with some fixup phase thing.

illegalprime · 2019-02-19T16:31:40Z

should armv7l-unknown-linux-gnueabihf be mapped to armv7-unknown-linux-gnueabihf (since that's the triple rust supports)?
also re: target/release, is it just a matter of adding:

"mv ${releaseDir} target/"

after cargo build?
this looks great BTW, can't wait!

matthewbauer · 2019-02-21T01:41:03Z

pkgs/build-support/rust/default.nix

@@ -36,21 +37,22 @@ let
      cargoDepsCopy="$sourceRoot/${cargoVendorDir}"
    '';

+  ccForBuild="${buildPackages.stdenv.cc}/bin/${buildPackages.stdenv.cc.targetPrefix}cc";


The build cc is almost never prefixed. I suppose there could be some weird case where it needs to be though.

illegalprime · 2019-02-22T06:23:27Z

I rebased this on the latest nixos-unstable and did the renaming for target/release as a postBuild hook, maybe it'll help:
fa82ebc...illegalprime:rust-cross
still testing but fd seems to build, ripgrep will build after #54510 i believe

Mic92 · 2019-02-22T08:17:59Z

@illegalprime can you just open a new pull request?
Then we can continue from there.

Mic92 · 2019-02-22T08:19:01Z

We can get rid of $releaseDir in this case I think:
fa82ebc...illegalprime:rust-crossdiff-2c7a7586fcf2d0d0d778bdddcf893a1bR124

Mic92 · 2019-02-22T08:20:23Z

and because people might have overwritten postBuild this could be moved to buildPhase itself.

illegalprime · 2019-02-22T20:02:52Z

will do! I believe pkgs.pkgsCross.aarch64-multiplatform.glibc is broken on staging so I'm trying to bisect. i'll post a PR when i have some concrete information on what works or not

Mic92 · 2019-02-22T22:10:37Z

You don't need to bisect. This was a commit where I introduced C.UTF-8: #56143 (comment)
The build process of this locale puts a non-native libc into the library path unfortunately.

vcunat · 2019-02-22T22:29:22Z

We could temporarily drop the new locale when cross-compiling or something like that, right? EDIT: I mean, we worked without it until recently, so it shouldn't be too bad...

Mic92 · 2019-02-22T22:31:15Z

@vcunat that might work also it would break other packages that assumes we have it, which might be python packages rather soon.

Mic92 · 2019-02-22T22:32:48Z

the problem is $(rtld-prefix) in the glibc makefile, which contains the host ld binary and is prefixed in the locale generation.

Mic92 · 2019-02-22T22:37:08Z

It is not too complicated to call localedef manually:
http://www.linuxfromscratch.org/clfs/view/clfs-2.0/arm/cross-tools/glibc.html
They seems to differentiate between endianness though by a flag, which we can query from our stdenv too.

illegalprime · 2019-02-22T22:38:39Z

thanks for linking that @Mic92!

bisecting is weird, maybe I don't understand it enough or all of the merge commits are making it complicated but this didn't find the right commit:

git bisect start nixpkgs/staging nixpkgs-channels/nixos-unstable
git bisect run nix build -f . pkgs.pkgsCross.aarch64-multiplatform.glibc

however this did:

git bisect start nixpkgs/staging HEAD~600
git bisect run nix build -f . pkgs.pkgsCross.aarch64-multiplatform.glibc

picking 600 commits in the past was just arbitrary and I'm lucky it found the commit , but how should one do this bisect in the future? do i have to flatten out the history to do it right or is my understanding just poor?

vcunat · 2019-02-23T08:21:05Z

Are you sure that nixpkgs-channels/nixos-unstable was a good commit (at that time)? I had bisected across merges many times and haven't noticed any issue with the algorithm, except that merges complicate the whole concept.

vcunat · 2019-02-23T08:24:47Z

it would break other packages that assumes we have it

Well, better than breaking glibc build already :-) It's why I meant it as a temporary workaround. The "assumes we have it" part won't be portable to non-glibc anyway, but perhaps those don't need it or something...

Mic92 · 2019-02-23T09:27:23Z

@vcunat this should be the proper fix instead: #56235

Mic92 · 2019-03-13T11:32:33Z

Superseded by #56540

Mic92 requested a review from LnL7 as a code owner Nov 21, 2018

Mic92 changed the title ~~Rust cross compilation~~ [WIP] Rust cross compilation Nov 21, 2018

GrahamcOfBorg added 6.topic: rust 8.has: documentation 10.rebuild-darwin-stdenv 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ labels Nov 21, 2018

matthewbauer reviewed Nov 21, 2018

View changes

pkgs/development/libraries/libpfm/default.nix Outdated Show resolved Hide resolved

matthewbauer reviewed Nov 21, 2018

View changes

Mic92 force-pushed the Mic92:rust-cross branch Nov 21, 2018

Mic92 force-pushed the Mic92:rust-cross branch Nov 25, 2018

GrahamcOfBorg removed the 10.rebuild-darwin-stdenv label Nov 25, 2018

Mic92 force-pushed the Mic92:rust-cross branch Nov 25, 2018

Mic92 mentioned this pull request Nov 25, 2018

Call for Content: 2018/13 NixOS/nixos-weekly#71

Merged

Mic92 force-pushed the Mic92:rust-cross branch Nov 26, 2018

GrahamcOfBorg added the 8.has: package (new) label Nov 26, 2018

Mic92 force-pushed the Mic92:rust-cross branch Nov 28, 2018

Mic92 mentioned this pull request Nov 28, 2018

[WIP] Port switch-to-configuration script to Python #48378

Closed

0 of 9 tasks complete

Ericson2314 reviewed Nov 28, 2018

View changes

pkgs/build-support/rust/default.nix Outdated Show resolved Hide resolved

Ericson2314 reviewed Nov 28, 2018

View changes

Mic92 force-pushed the Mic92:rust-cross branch Dec 18, 2018

rust: fix cross-compilation

Loading status checks…

bceda58

Mic92 force-pushed the Mic92:rust-cross branch to bceda58 Dec 18, 2018

Mic92 reviewed Dec 18, 2018

View changes

matthewbauer reviewed Feb 21, 2019

View changes

ryantm added the 2.status: work-in-progress label Feb 24, 2019

illegalprime mentioned this pull request Feb 28, 2019

Rust Cross Compilation (1.32.0) #56540

Merged

17 of 24 tasks complete

Mic92 closed this Mar 13, 2019

Mic92 deleted the Mic92:rust-cross branch Mar 13, 2019

NixOS / nixpkgs

Sponsor NixOS/nixpkgs

[WIP] Rust cross compilation #50866

[WIP] Rust cross compilation #50866

Conversation

Mic92 commented Nov 21, 2018 • edited

Status

Things done

Mic92 commented Nov 21, 2018 • edited

GrahamcOfBorg commented Nov 21, 2018

GrahamcOfBorg commented Nov 21, 2018

This comment has been minimized.

matthewbauer Nov 21, 2018 Member

This comment has been minimized.

Mic92 Nov 21, 2018 Author Contributor

This comment has been minimized.

Ericson2314 Nov 28, 2018 Member

This comment has been minimized.

matthewbauer Nov 21, 2018 Member

This comment has been minimized.

Mic92 Nov 21, 2018 Author Contributor

This comment has been minimized.

Mic92 Nov 21, 2018 Author Contributor

This comment has been minimized.

LnL7 Nov 21, 2018 Member

This comment has been minimized.

Mic92 Nov 21, 2018 Author Contributor

GrahamcOfBorg commented Nov 21, 2018

Mic92 commented Nov 21, 2018

Mic92 commented Nov 21, 2018

Mic92 commented Nov 25, 2018

Mic92 commented Nov 28, 2018

This comment has been minimized.

Ericson2314 Nov 28, 2018 Member

This comment has been minimized.

Mic92 Nov 29, 2018 Author Contributor

This comment has been minimized.

Mic92 Dec 18, 2018 Author Contributor

This comment has been minimized.

Ericson2314 Dec 18, 2018 Member

Ericson2314 commented Jan 2, 2019

Mic92 commented Jan 2, 2019

Ericson2314 commented Jan 2, 2019

Mic92 commented Jan 2, 2019 • edited

Ericson2314 commented Jan 18, 2019

illegalprime commented Feb 19, 2019

This comment has been minimized.

matthewbauer Feb 21, 2019 Member

illegalprime commented Feb 22, 2019 • edited

Mic92 commented Feb 22, 2019

Mic92 commented Feb 22, 2019

Mic92 commented Feb 22, 2019

illegalprime commented Feb 22, 2019

Mic92 commented Feb 22, 2019

vcunat commented Feb 22, 2019 • edited

Mic92 commented Feb 22, 2019

Mic92 commented Feb 22, 2019

Mic92 commented Feb 22, 2019

illegalprime commented Feb 22, 2019

vcunat commented Feb 23, 2019

vcunat commented Feb 23, 2019

Mic92 commented Feb 23, 2019

Mic92 commented Mar 13, 2019

Mic92 commented Nov 21, 2018 •

edited

Mic92 commented Nov 21, 2018 •

edited

matthewbauer Nov 21, 2018
Member

Mic92 Nov 21, 2018
Author Contributor

Ericson2314 Nov 28, 2018
Member

matthewbauer Nov 21, 2018
Member

Mic92 Nov 21, 2018
Author Contributor

Mic92 Nov 21, 2018
Author Contributor

LnL7 Nov 21, 2018
Member

Mic92 Nov 21, 2018
Author Contributor

Ericson2314 Nov 28, 2018
Member

Mic92 Nov 29, 2018
Author Contributor

Mic92 Dec 18, 2018
Author Contributor

Ericson2314 Dec 18, 2018
Member

Mic92 commented Jan 2, 2019 •

edited

matthewbauer Feb 21, 2019
Member

illegalprime commented Feb 22, 2019 •

edited

vcunat commented Feb 22, 2019 •

edited