Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rust Cross Compilation (1.32.0) #56540

Merged
merged 11 commits into from Apr 1, 2019

Conversation

@illegalprime
Copy link
Member

commented Feb 28, 2019

Motivation for this change

This is the same as #50866 but re-based and updated for Rust 1.32.0. It also does some other extra stuff like moving all output dirs to target/{release, debug} and being able to specify the build mode (release or debug).

the following derivations build:

  • pkgs.pkgsCross.aarch64-multiplatform.rustc
  • pkgs.pkgsCross.aarch64-multiplatform.cargo
  • pkgs.pkgsCross.aarch64-multiplatform.fd
  • pkgs.pkgsCross.aarch64-multiplatform.synapse-bt
  • pkgs.pkgsCross.aarch64-multiplatform.webmetro
  • pkgs.pkgsCross.aarch64-multiplatform.miniserve
  • pkgs.pkgsCross.aarch64-multiplatform.ripgrep
  • pkgs.pkgsCross.aarch64-multiplatform.ethabi
  • pkgs.pkgsCross.aarch64-multiplatform.zola
  • pkgs.pkgsCross.aarch64-multiplatform.todiff
  • pkgs.pkgsCross.aarch64-multiplatform.gitAndTools.git-ignore
  • pkgs.pkgsCross.aarch64-multiplatform.gitAndTools.git-codeowners
  • pkgs.pkgsCross.aarch64-multiplatform.gitAndTools.git-absorb
  • pkgs.rustc

I will attempt to updated this as soon as I figure out how to build more derivations. There are a lot more rust packages in nixpkgs, but these are likely a good starting point.

Any edits to or help with these derivations is appreciated.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@illegalprime

This comment has been minimized.

Copy link
Member Author

commented Mar 11, 2019

not sure why, maybe @matthewbauer might know, but I get these errors on a lot of rust packages: ripgrep, zola, etc.

/nix/store/hxwlbcm8bbhdacayc90jljcd76w434r0-aarch64-unknown-linux-gnu-binutils-2.31.1/bin/aarch64-unknown-linux-gnu-ld: /nix/store/f6lrlfbg9ddvvw20j6av4ym7hxy6kxx1-rustc-1.32.0/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-e27907af0047d202.rlib(std-e27907af0047d202.std.3x3nkc5s-cgu.0.rcgu.o): Relocations in generic ELF (EM: 0)
/nix/store/hxwlbcm8bbhdacayc90jljcd76w434r0-aarch64-unknown-linux-gnu-binutils-2.31.1/bin/aarch64-unknown-linux-gnu-ld: /nix/store/f6lrlfbg9ddvvw20j6av4ym7hxy6kxx1-rustc-1.32.0/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-e27907af0047d202.rlib: error adding symbols: file in wrong format
          collect2: error: ld returned 1 exit status

it seems like libstd is in the wrong format, even though rust seems to think it's right, (because it placed it in the aarch64-unknown-linux-gnu folder). so maybe rustc is being build incorrectly.

but the fd package seems to build fine, it's the exception however. i believe it's not using the same libstd somehow, but i know it's using the same rustc

UPDATE: fixed with a weird hack bd45c09 dunno what 2 do about that

@Mic92

This comment has been minimized.

Copy link
Contributor

commented Mar 12, 2019

I have not looked yet at the rlib format, but I read it is a elf file with some extras, so using readelf should provide you some information.

@illegalprime

This comment has been minimized.

Copy link
Member Author

commented Mar 12, 2019

@Mic92 yes, (see list above) cross-compiled cargo, rustc, and other packages I've tested are working! The only thing that isn't "working" is compiling rustc to be a cross-compiler that runs on the build platform and has cross-compiled host libs. I get around it with the hack I mentioned earlier.

the right architecture lib seems to get compiled, but there's something wrong with them. :/

EDIT: Specificity
The error I described above is a linking error with the first object inside of the rlib (since the offset is 0 bytes), using readelf -h we get:

File: /nix/store/qzpw6c9i8jk3lq3wbaspawc7mcwslryz-my-rustc/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-e27907af0047d202.rlib(std-e
27907af0047d202.std.3x3nkc5s-cgu.0.rcgu.o)
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           None
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          153720 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         558
  Section header string table index: 557

The same from the cross-compiled rustc:

File: /nix/store/7a4xsgvjzk1j49l7vp17szg3yfqqqhcb-rustc-1.32.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libs
td-3367fd61bbaa28f9.rlib(std-3367fd61bbaa28f9.std.d5c46zzr-cgu.0.rcgu.o)
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          154192 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         560
  Section header string table index: 559

The only thing I can guess is that the Machine field is None in the bad rlib, but as you can see both of them look like they're the right architecture (and are placed in the correct location for rustc to pick them up as that architecture)

@Mic92

This comment has been minimized.

Copy link
Contributor

commented Mar 12, 2019

Can you upload? /nix/store/qzpw6c9i8jk3lq3wbaspawc7mcwslryz-my-rustc/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-e27907af0047d202.rlib the one with the broken Machine field.


cargoBuildFlags = [ "--all" ];

PKG_CONFIG_ALLOW_CROSS = 1;

This comment has been minimized.

Copy link
@Mic92

Mic92 Mar 12, 2019

Contributor

Is this something we should set by default in buildRustPackage?

This comment has been minimized.

Copy link
@illegalprime

illegalprime Mar 12, 2019

Author Member

possibly, it's for pkg-config so ostensibly it would need to be set anywhere pkg-config is used during cross-compilation. Don't know how to handle all those cases though.

This comment has been minimized.

Copy link
@matthewbauer

matthewbauer Mar 12, 2019

Member

Actually it is a pkgconfig-rs thing i think

This comment has been minimized.

Copy link
@Mic92

Mic92 Mar 13, 2019

Contributor

Maybe some hook in buildRustPackage that checks if pkg-config is available and sets PKG_CONFIG_ALLOW_CROSS in that case?

@illegalprime

This comment has been minimized.

Copy link
Member Author

commented Mar 12, 2019

@Mic92 so this happens when you build rustc with buildPlatform == hostPlatform != targetPlatform, here you go:
libstd-e27907af0047d202.zip

@Mic92

This comment has been minimized.

Copy link
Contributor

commented Mar 12, 2019

Looks actually sane on the first glance:

$ 7z x libstd-e27907af0047d202.zip
$ ar x libstd-e27907af0047d202.rlib
$ file *.o
alloc.o:                                        ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
backtrace.o:                                    ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
dwarf.o:                                        ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
elf.o:                                          ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
fileline.o:                                     ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
posix.o:                                        ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
read.o:                                         ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
sort.o:                                         ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
state.o:                                        ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.0.rcgu.o: ELF 64-bit LSB relocatable, no machine, version 1 (SYSV), not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.1.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.2.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.3.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.4.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.5.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.6.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.7.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.8.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.9.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
@illegalprime

This comment has been minimized.

Copy link
Member Author

commented Mar 12, 2019

@Mic92 yea, could setting the Machine correctly really be all that's needed?

@Mic92

This comment has been minimized.

Copy link
Contributor

commented Mar 12, 2019

@illegalprime could be. I have not figured out yet, how I can run gdb on the build compiler otherwise I would set a breakpoint in the right location. The hack is ok right now, also annoying because it forces cross-compilers to build a rustc they probably never need. Maybe this can be reported upstream though. I wonder how they build those stdlibs.

rm -rf $out/lib/rustlib/${stdenv.hostPlatform.config}
cp -a \
${pkgsCross."${host}".rustc}/lib/rustlib/${stdenv.hostPlatform.config}/. \

This comment has been minimized.

Copy link
@matthewbauer

matthewbauer Mar 12, 2019

Member

This should be targetpackages - pkgsCross should not be referenced internally.

This comment has been minimized.

Copy link
@illegalprime

illegalprime Mar 13, 2019

Author Member

if i try to use targetPackages.rustc I get:
error: attribute 'rustc' missing,

This comment has been minimized.

Copy link
@Ericson2314

Ericson2314 Mar 13, 2019

Member

Shouldn't it be buildPackages? Or maybe just pkgs? I'm a bit confused on what the intention is.

This comment has been minimized.

Copy link
@illegalprime

illegalprime Mar 13, 2019

Author Member

this rustc is supposed to be the rustc whose host & target is aarch64

This comment has been minimized.

Copy link
@illegalprime

illegalprime Mar 13, 2019

Author Member

I think pkgs.rustc is where just the target is aarch64 and build & host are x86


cargoBuildFlags = [ "--all" ];

PKG_CONFIG_ALLOW_CROSS = 1;

This comment has been minimized.

Copy link
@matthewbauer

matthewbauer Mar 12, 2019

Member

Actually it is a pkgconfig-rs thing i think

ccForBuild = "${buildPackages.stdenv.cc}/bin/${buildPackages.stdenv.cc.targetPrefix}cc";
cxxForBuild = "${buildPackages.stdenv.cc}/bin/${buildPackages.stdenv.cc.targetPrefix}c++";
ccForTarget = "${targetPackages.stdenv.cc}/bin/${targetPackages.stdenv.cc.targetPrefix}cc";
cxxForTarget = "${targetPackages.stdenv.cc}/bin/${targetPackages.stdenv.cc.targetPrefix}c++";

This comment has been minimized.

Copy link
@matthewbauer

matthewbauer Mar 12, 2019

Member

This looks wrong. /cc @Ericson2314

@Mic92 Mic92 referenced this pull request Mar 13, 2019
3 of 13 tasks complete
@illegalprime

This comment has been minimized.

Copy link
Member Author

commented Mar 13, 2019

turns out it was the stripping in the fixupPhase that was creating the bad rlibs, not rust's bootstrapping process, :(

@Mic92

This comment has been minimized.

Copy link
Contributor

commented Mar 13, 2019

Good news. Anything left to do then?

@illegalprime

This comment has been minimized.

Copy link
Member Author

commented Mar 13, 2019

@Mic92 i have an interest in cross compiling for 32-bit arm (armv7), doing:

nix build -f . pkgsCross.armv7l-hf-multiplatform.ripgrep

breaks because Rust doesn't recognize the triple armv7l-unknown-linux-gnueabihf, but is does recognize armv7-unknown-linux-gnueabihf (no l). my whole wanting rust-cross-compilation is for this target so I'd like to be able to get that working before this merges, should I:

  1. somewhere within buildRustPackage take that triple and remove the l when giving it to rust
  2. add a new entry in pkgsCross
  3. ???

if you think no. 2 is preferable, how do I do that? the entire layout of pkgsCross is a mystery to me.

EDIT: less of a mystery, after knowing it's in pkgs/top-level/stage.nix. I added my own target but had to add this as a cpu type:

diff --git a/lib/systems/parse.nix b/lib/systems/parse.nix
index 6947d41419e..c029d6a1914 100644
--- a/lib/systems/parse.nix
+++ b/lib/systems/parse.nix
@@ -76,6 +76,7 @@ rec {
     armv7r   = { bits = 32; significantByte = littleEndian; family = "arm"; version = "7"; };
     armv7m   = { bits = 32; significantByte = littleEndian; family = "arm"; version = "7"; };
     armv7l   = { bits = 32; significantByte = littleEndian; family = "arm"; version = "7"; };
+    armv7    = { bits = 32; significantByte = littleEndian; family = "arm"; version = "7"; };
     armv8a   = { bits = 32; significantByte = littleEndian; family = "arm"; version = "8"; };
@illegalprime

This comment has been minimized.

Copy link
Member Author

commented Mar 13, 2019

another thing since we're here, how do people feel about https://www.hadean.com/blog/managing-rust-dependencies-with-nix-part-ii (i'm not the author, or know the author)? It seems more nix-ish than the current setup, but it didn't seem very popular in #11144.

my other main goal is to be able to support rust builds where some crates are in directories that are siblings to the project directory, without setting the entire parent folder as my src. this is because there's a lot of other stuff in that folder as well (that are dependencies for different project altogether)

@illegalprime illegalprime changed the title [WIP] Rust cross compilation (1.32.0) Rust Cross Compilation (1.32.0) Mar 13, 2019
@Ericson2314

This comment has been minimized.

Copy link
Member

commented Mar 13, 2019

I rebased @Mic92's old PR and changed some things in the past. I'll work to get that rebased so I can combine with this.

@Ericson2314

This comment has been minimized.

Copy link
Member

commented Mar 13, 2019

https://github.com/NixOS/nixpkgs/pull/57611/files is something that would make this easier.

@Ericson2314

This comment has been minimized.

Copy link
Member

commented Mar 14, 2019

https://github.com/Ericson2314/nixpkgs/branche/rust-cross here my old modification, rebased to have the same branch as this, but not yet tested). However, if #57611 is merged, we should rebase both onto that before combining.

@illegalprime

This comment has been minimized.

Copy link
Member Author

commented Mar 14, 2019

After some hacking, ignore my previous 2 comments, I'm all good on this PR I think it's ready to merge! 🎉

] ++ optional (!withBundledLLVM) [
"--enable-llvm-link-shared"
"${setBuild}.llvm-config=${buildLlvmShared}/bin/llvm-config"
"${setHost}.llvm-config=${buildLlvmShared}/bin/llvm-config"

This comment has been minimized.

Copy link
@illegalprime

illegalprime Mar 16, 2019

Author Member

this might be wrong, need to test

This comment has been minimized.

Copy link
@illegalprime

illegalprime Mar 16, 2019

Author Member

so when building under ARMv7 rust tries to call llvm-config to get the libs to link against. buildLlvmShared has an llvm-config that can run on the build system but the wrong shared libraries, while llvm-config is the opposite (wrong binary, right libs). Not sure why this problem doesn't show up on AArch64.

This comment has been minimized.

Copy link
@illegalprime

This comment has been minimized.

Copy link
@Mic92

Mic92 Mar 20, 2019

Contributor

Since there should be no arch-specific one could wrap around the llvm-config
and then just do sed -e "s!${buildPackages.llvm}!${llvm}!" on the output?

@illegalprime illegalprime referenced this pull request Mar 17, 2019
2 of 10 tasks complete
@illegalprime

This comment has been minimized.

Copy link
Member Author

commented Mar 25, 2019

@Mic92 NP, do you have documented somewhere the philosophy and plan for nixpkgs cross-comp? things like depsTargetTargetPropagated and that PR make me feel like there's a larger architecture goal at play that I don't quite understand.

@Ericson2314

This comment has been minimized.

Copy link
Member

commented Mar 26, 2019

I assume you meant me? :). So https://nixos.org/nixpkgs/manual/#ssec-stdenv-dependencies is the main section describing the dependencies and how they fit together, but in that PR I extended the corresponding section the cross compilation section. Hopefully https://hydra.nixos.org/build/91228503# has it in a moment.

edit master post PR built:

@Ericson2314

This comment has been minimized.

Copy link
Member

commented Mar 26, 2019

OK now that that #57611 is merged:

The first thing to note is @Mic92 did take my changes (all-packages.nix changes were moved to an I guess orphaned default.nix) I missed that before, sorry. This means there will be little-to-nothing to double check against my branch :).

The second things is @illegalprime your PR is harder to review because some commits "undo" other commits (revert or moral revert). Could you git rebase -i so the commits tell a nicer, if artificial, "story"? (Feel free to start with the rebase I did, which squashed the "remove jemalloc" change since the original commit added it back only after an prior rebase's conflict resolution.)

After that's done I'll be able to better wrap my head around everything, and also set about incorporating the new pkgsFooBar ewnly-added in #57611, which will help with cross compiling rustc (build != host == target).

Thanks in advance!

@illegalprime illegalprime force-pushed the illegalprime:rust-cross branch from 2d6d180 to 6d06924 Mar 27, 2019
@illegalprime

This comment has been minimized.

Copy link
Member Author

commented Mar 27, 2019

thanks for the links to the docs @Mic92 , i rebased it, we'll see how the tests go

@Ericson2314 Ericson2314 force-pushed the illegalprime:rust-cross branch from 6d06924 to 2a67071 Mar 29, 2019
@Ericson2314 Ericson2314 force-pushed the illegalprime:rust-cross branch from 2a67071 to 3bcede8 Mar 29, 2019
Copy link
Member

left a comment

Thanks again @illegalprime and @Mic92! I pruned the remaining silly states from the history, and incorporated the new pkgsBuild*, so this is ready to go!

@Ericson2314

This comment has been minimized.

Copy link
Member

commented Mar 29, 2019

https://github.com/Ericson2314/nixpkgs/commits/rust-cross-2 has the beginnings of a buildRustCrate, but that can be merged later.

@Ericson2314 Ericson2314 changed the base branch from master to staging Apr 1, 2019
@Ericson2314 Ericson2314 merged commit b621207 into NixOS:staging Apr 1, 2019
12 of 13 checks passed
12 of 13 checks passed
cargo, librsvg, synapse-bt on x86_64-linux Timed out, unknown build status
Details
cargo, librsvg, synapse-bt on aarch64-linux Success
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@Ericson2314

This comment has been minimized.

Copy link
Member

commented Apr 1, 2019

NixOS/ofborg looked good enough to me, haha.

@illegalprime illegalprime referenced this pull request Apr 23, 2019
6 of 14 tasks complete
@c0bw3b c0bw3b referenced this pull request May 22, 2019
4 of 10 tasks complete
@arcnmx arcnmx referenced this pull request Jun 22, 2019
3 of 10 tasks complete
@AtkinsChang AtkinsChang referenced this pull request Jun 26, 2019
4 of 10 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.