Improve cross-compilation example

[lovesegfault]

Currently, the cross-compilation example instantiates nixpkgs like so:

crane/examples/custom-toolchain/flake.nix

Lines 23 to 26 in 5afc3e9

	pkgs = import nixpkgs {
	inherit system;
	overlays = [ (import rust-overlay) ];
	};

And proceeds as usual, only referencing the custom toolchain when instantiating craneLib:

crane/examples/custom-toolchain/flake.nix

Lines 28 to 40 in 5afc3e9

	rustWithWasiTarget = pkgs.rust-bin.stable.latest.default.override {
	targets = [ "wasm32-wasi" ];
	};
	# NB: we don't need to overlay our custom toolchain for the *entire*
	# pkgs (which would require rebuidling anything else which uses rust).
	# Instead, we just want to update the scope that crane will use by appending
	# our specific toolchain there.
	craneLib = (crane.mkLib pkgs).overrideScope' (final: prev: {
	rustc = rustWithWasiTarget;
	cargo = rustWithWasiTarget;
	rustfmt = rustWithWasiTarget;
	});

This is AOK if you don't have any system dependencies, but it would be great if an example were added showing how to use crane together with crossSystem, akin to the example in rust-overlay: https://github.com/oxalica/rust-overlay/blob/master/examples/cross-aarch64/shell.nix.

I've attempted to get this to work with the following (truncated) snippet:

{
        localSystem = "x86_64-linux";
        crossSystem = "aarch64-linux";
        pkgs = import nixpkgs {
          inherit localSystem crossSystem;
          overlays = [ rust-overlay.overlay ];
        };

        rustToolchain = pkgs.pkgsBuildHost.rust-bin.stable.latest.default;

        craneLib = (crane.mkLib pkgs).overrideScope' (final: prev: {
          cargo = rustToolchain;
          rustc = rustToolchain;
          clippy = rustToolchain;
          rustfmt = rustToolchain;
        });

        src =  ./.;

        cargoArtifacts = craneLib.buildDepsOnly {
          inherit src;
        };

        crateFmt = craneLib.cargoFmt {
          inherit cargoArtifacts src;
        };

        crateClippy = craneLib.cargoClippy {
          inherit src;
          cargoArtifacts = crateFmt;
          cargoClippyExtraArgs = "-- --deny warnings";
        };

        crate = craneLib.buildPackage {
          inherit src;
          cargoArtifacts = crateClippy;
        };
}

But it fails with the following:

hyperpixel_init-clippy-aarch64-unknown-linux-gnu> unable to copy cargo artifacts, "/nix/store/ppzyixshipwzr7zqvq190mdfbcap6njg-hyperpixel_init-fmt-aarch64-unknown-linux-gnu-0.1.0/target.tar.zst" looks invalid

You should be able to reproduce this with nix build -L on https://github.com/lovesegfault/hyperpixel-init on the crane branch.

[ipetkov]

Thanks for the detailed config, definitely helps reproduce things locally!

I think the root of the issue is the fact that cargoFmt does not produce any artifacts, so chaining things to enforce that formatting is run, then clippy, then build breaks.

crane/lib/cargoFmt.nix

Line 16 in 5afc3e9

doInstallCargoArtifacts = false;

My original assumption was that we don't want to generate artifacts for a cargo fmt run since there is no point in copying around the artifacts a second time (or trying to upload them to the cache...).

On one hand we can make this configurable (default to not installing cargo artifacts but allow the caller to specify they want to), but it could be a discoverability issue. On the other hand, we can let the artifacts be installed by default, but it would waste cycles for most projects which don't know about the nitty gritty details...

[lovesegfault]

I think the root of the issue is the fact that cargoFmt does not produce any artifacts, so chaining things to enforce that formatting is run, then clippy, then build breaks.

Ah, seems like I was being bitten by an issue unrelated to cross-compiling, oops!

I removed the crateFmt portion of the code (the aforementioned branch is updated), so now a new failure happens:

hyperpixel_init-deps-aarch64-unknown-linux-gnu>     Finished release [optimized] target(s) in 7.31s
hyperpixel_init-deps-aarch64-unknown-linux-gnu> installing
hyperpixel_init-deps-aarch64-unknown-linux-gnu> running: mkdir -p $out
hyperpixel_init-deps-aarch64-unknown-linux-gnu> copying target to /nix/store/cclsh28vgx7x2lwmci9wgdhwdaajah3k-hyperpixel_init-deps-aarch64-unknown-linux-gnu-0.1.0/target.tar.zst
hyperpixel_init-deps-aarch64-unknown-linux-gnu> /nix/store/j32xp13h5js4958kl28nqa5y8s82wyv6-installCargoArtifactsHook/nix-support/setup-hook: line 20: /nix/store/gwml59z3d1pvqngv1wzjz4884vyva01b-zstd-aarch64-unknown-linux-gnu-1.5.1-bin/bin/zstd: cannot execute binary file: Exec format error
hyperpixel_init-deps-aarch64-unknown-linux-gnu> tar: -: Wrote only 4096 of 10240 bytes
hyperpixel_init-deps-aarch64-unknown-linux-gnu> tar: Error is not recoverable: exiting now

I suspect crane is using pkgs.zstd where what it wants is pkgs.pkgsBuildBuild.zstd (c.f. https://github.com/NixOS/nixpkgs/blob/master/doc/stdenv/cross-compilation.chapter.md)

[ipetkov]

@lovesegfault if you're interested in adding a cross-compilation example I'd be happy to merge it in! Otherwise I'll try to add one when I get some free time

[lovesegfault]

@lovesegfault if you're interested in adding a cross-compilation example I'd be happy to merge it in! Otherwise I'll try to add one when I get some free time

I'm working on it, but I haven't been able to get it to work yet. One issue I'm having is that it seems to always want to compile rustc, even if you use .overrideScope' which is a problem since that takes hours.

You can check this by doing a nix-build in that repo I linked earlier (I updated the branch again).

I haven't been able to understand why it always tries to cross-compile rustc.

[lovesegfault]

The more I think about this, the more certain I am that something is wrong here:

crane/flake.nix

Lines 18 to 24 in 0125041

	myPkgsFor = pkgs: pkgs.callPackages ./pkgs { };
	mkLib = pkgs: import ./lib {
	inherit (nix-std.lib.serde) fromTOML toTOML;
	inherit (pkgs) lib newScope;
	myPkgs = myPkgsFor pkgs;
	};

I haven't been able to figure out the issue though.

[lovesegfault]

I placed a complete reproducer here: https://github.com/lovesegfault/crane-cross-example

It's really interesting that nix build .\#cargoArtifacts and nix build .\#crateClippy work, but building the package (nix build .\#crate) doesn't.

Suggest something about buildPackage is funky.

[lovesegfault]

crane/lib/mkCargoDerivation.nix

Lines 51 to 55 in 0125041

	configureCargoCommonVarsHook
	configureCargoVendoredDepsHook
	inheritCargoArtifactsHook
	installCargoArtifactsHook
	remapSourcePathPrefixHook

I wonder whether these should come from pkgsBuildBuild explicitly.

I still feel like the overrideScope' is not being applied somewhere though.

[ipetkov]

The more I think about this, the more certain I am that something is wrong here

It's entirely possible, I'll admit I wasn't thinking about cross compilation at all 😅
I'm all for refactoring the internals if there's a more correct way of specifying stuff!

I wonder whether these should come from pkgsBuildBuild explicitly.

Probably shouldn't matter for the shell scripts (but could be good for consistency). Wonder where cargo needs to be imported from?

Technically the rust compiler is a first-class cross compiler (as long as you have the right targets imported), but not sure how it plays out with the versions that are available in nixpkgs. Maybe cargo needs to be pkgsBuildHost? Not entirely sure

[ipetkov]

I've been (re)reading through the cross compilation section of the manual to better understand the correct way to configure expressions. It sounds like thanks to some package splicing logic things "should" work as long as dependencies are correctly scoped in their dependency types (e.g. depsBuildBuild, nativeBuildInputs (i.e. depsBuildHost), etc.).

I'll play around with the repro example when I get a chance and try to see what could be suspect there. One thing which comes to mind is the fact that crane will specify nativeBuildInputs = [ cargo ] (the cargo attribute implicitly wraps the rustc being invoked). Perhaps this needs to be scoped differently (buildDepsTarget maybe? I need to wrap my head around this a bit more) so that Nix doesn't end up building something redundantly

[ipetkov]

Figured out the issue, it had to do with pkg splicing not happening automatically. Should have a fix in sometime this week!