Running cross and cargo builds for different targets breaks incremental compilation

[ryankurte]

When building a project for a couple of different targets, one within cross and one outside, swapping targets seems to cause the whole project to rebuild. I'm not entirely sure of the mechanism, but, suspect it is either related to changes in the profile (per rust-lang/cargo#4961) or different compiler versions. This would seem to imply the profile is project-global rather than per-target and I haven't dug into this at all.

In case anyone else runs into this, if you only have one target under cross you can work around the issue by placing the project in a workspace then building natively (via cargo build) from the workspace folder and via cross in the project folder (as the docker container cannot see up to the workspace directory) so that the target directories are split, in which case incremental builds work properly. If you need more than one cross target, I'm not sure there's a great option, maybe propagating sccache config / files into the cross container :-/

[nui]

In case anyone else runs into this, I you only have one target under cross you can workaround this by placing the project in a workspace then building natively (via cargo build) from the workspace folder and via cross in the project folder (as the docker container cannot see up to the workspace directory) so that the target directories are split, in which case incremental builds work properly. If you need more than one cross target, I'm not sure there's a great option, maybe propagating sccache config / files into the cross container :-/

Another workaround is to use custom target directory (--target-dir) when build with cross

[Alexhuszagh]

Note this only affects images with build scripts, I believe. I've tried compiling in the two following cases:

$ git clone https://github.com/cross-rs/rust-cpp-hello-word
$ cd rust-cpp-hello-word
$ cross run --target x86_64-unknown-linux-gnu
   Compiling cc v1.0.73
   Compiling hellopp v0.1.0 (/project)
    Finished dev [unoptimized + debuginfo] target(s) in 2.48s
     Running `/linux-runner x86_64 /target/x86_64-unknown-linux-gnu/debug/hellopp`
Hello, world!
$ cross run --target armv7-unknown-linux-gnueabihf
   Compiling hellopp v0.1.0 (/project)
    Finished dev [unoptimized + debuginfo] target(s) in 0.51s
     Running `/linux-runner armv7 /target/armv7-unknown-linux-gnueabihf/debug/hellopp`
Hello, world!
$ cargo run
   Compiling cc v1.0.73
   Compiling hellopp v0.1.0 (/home/ahuszagh/Desktop/cross/rust-cpp-hello-word)
    Finished dev [unoptimized + debuginfo] target(s) in 2.71s
     Running `target/debug/hellopp`
Hello, world!
$ cross run --target armv7-unknown-linux-gnueabihf
   Compiling cc v1.0.73
   Compiling hellopp v0.1.0 (/project)
    Finished dev [unoptimized + debuginfo] target(s) in 2.07s
     Running `/linux-runner armv7 /target/armv7-unknown-linux-gnueabihf/debug/hellopp`
Hello, world!
$ cross run --target x86_64-unknown-linux-gnu
   Compiling hellopp v0.1.0 (/project)
error: failed to run custom build command for `hellopp v0.1.0 (/project)`

Caused by:
  process didn't exit successfully: `/target/debug/build/hellopp-3a1e8134447a8a4c/build-script-build` (exit status: 1)
  --- stderr
  /target/debug/build/hellopp-3a1e8134447a8a4c/build-script-build: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /target/debug/build/hellopp-3a1e8134447a8a4c/build-script-build)
$ cross run --target armv7-unknown-linux-gnueabihf
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `/linux-runner armv7 /target/armv7-unknown-linux-gnueabihf/debug/hellopp`
Hello, world!

Meanwhile, cargo init --bin hello and then using the same steps compiles fine. This also seems to have an issue with invalidating builds for the same target.

Note that with a build.rs dependency, it seems it only invalidates on the cross build after cargo build, but not after. cargo run and then cross run --target x86_64-unknown-linux-gnu re-builds the cc, but not if I add a cross run --target armv7-unknown-linux-gnueabihf in between. Using cross multiple times never invalidates builds, it seems.

[Alexhuszagh]

FYI: we've added documentation for sccache support, so this is at least done.

[Alexhuszagh]

This seems to happen only with build scripts, and it seems that the hash is partially based on the path of the toolchain and cargo itself, this might be related to how we mount at /cargo and not at the original path. This may not be that hard to fix.

Update: Ensuring the host and mount paths are the same works perfectly, even on WSL2 where the mount paths need to be converted to a WSL path. This is linked to rust-lang/cargo#4961.

[giggio]

Hey everyone, when is this expected to be released? It would help me clean up some scripts!

[Emilgardis]

There's no plan for a 0.2.5 release right now, we're aiming for 0.3.0, but the time table is not set.

For now, consider using cross from the main branch. If someone wants to implement 0.2.5 from https://github.com/cross-rs/cross/tree/v0.2.4 and only including #947 I'd be happy to do a release.

shouldn't be too hard