servo-linux-cross2 is having trouble building arm64

[jdm]

--- stderr
configure: error: in `/home/servo/buildbot/slave/arm64/build/target/release/build/backtrace-sys-204dc57c91e9a514/out':
configure: error: C compiler cannot create executables
See `config.log' for more details.
thread 'main' panicked at 'failed with: exit code: 77', /home/servo/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-sys-0.1.11/build.rs:136

[jdm]

Haven't seen this for a long time.

[jdm]

Saw this again on servo-linux-cross1 today in servo/servo#18821.

[SimonSapin]

This happened for both arm32 and arm64 in servo/servo#18821, both times on servo-linux-cross1.

[jdm]

Both arm32 and arm64 failed both times, and across all of servo-linux-cross1, servo-linux-cross2, and servo-linux-cross3. I wish I could get ahold of the config.log files that were generated for those builds :(

[jdm]

configure:2977: checking for C compiler default output file name
configure:2999: aarch64-linux-gnu-gcc -O3 -ffunction-sections -fdata-sections -fPIC -m64   conftest.c  >&5
aarch64-linux-gnu-gcc: error: unrecognized command line option '-m64'

[jdm]

The argument in question comes from https://github.com/alexcrichton/cc-rs/blob/a84c5ee84e1a18091dd8c46b45912567fde9dbda/src/lib.rs#L614-L619 but I'm still confused how this sometimes works and sometimes does not.

[jdm]

Apparently when the build succeeds, the command executed is this instead:

configure:2977: checking for C compiler default output file name
configure:2999: aarch64-linux-gnu-gcc -O3 -ffunction-sections -fdata-sections -fPIC   conftest.c  >&5

Specifically, the -m64 is missing. The failing log was retried from /home/servo/buildbot/slave/arm64/build/target/release/build/backtrace-sys-ef6211128d6f7e04/out/config.log, while the successful log was at /home/servo/buildbot/slave/arm64/build/target/aarch64-unknown-linux-gnu/release/build/backtrace-sys-87e91972833634cb/out/config.log. Note how one is build/target/release/build, while the other is build/target/aarch64-unknown-linux-gnu/release/build.

[jdm]

Accordingly, the two configure scripts are invoked slightly differently:
Failed:
/home/servo/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-sys-0.1.11/src/libbacktrace/configure --with-pic --disable-multilib --disable-shared --disable-host-shared --host=x86_64-unknown-linux-gnu --build=x86_64-unknown-linux-gnu

Successful:
/home/servo/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-sys-0.1.11/src/libbacktrace/configure --with-pic --disable-multilib --disable-shared --disable-host-shared --host=aarch64-unknown-linux-gnu --build=x86_64-unknown-linux-gnu

[jdm]

Cargo is definitely building backtrace-sys with an unexpected target, according to this output:

TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CC_x86_64-unknown-linux-gnu = None
CC_x86_64_unknown_linux_gnu = None
HOST_CC = None
CC = Some("aarch64-linux-gnu-gcc")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CFLAGS_x86_64-unknown-linux-gnu = None
CFLAGS_x86_64_unknown_linux_gnu = None
HOST_CFLAGS = None
CFLAGS = None
PROFILE = Some("release")

[SimonSapin]

CC @alexcrichton, in case you’ve seen this before or have some idea what’s going on.

[jdm]

Well, either it's an unexpected target, or it's expected but cc-rs is finding the wrong compiler for it (perhaps we're setting CC somehow before the backtrace-rs build script is invoked?).

[jdm]

Ok, so the compiler is being set to the cross binary by the build environment in

saltfs/buildbot/master/files/config/environments.py

Lines 113 to 144 in df07d3a

	build_arm32 = build_arm.without(['SERVO_CACHE_DIR']) + Environment({
	'BUILD_TARGET': 'arm-unknown-linux-gnueabihf',
	'CC': 'arm-linux-gnueabihf-gcc',
	'CXX': 'arm-linux-gnueabihf-g++',
	'PATH': ':'.join([
	'{{ common.servo_home }}/bin',
	'/usr/local/sbin',
	'/usr/local/bin',
	'/usr/bin',
	'/usr/sbin',
	'/sbin',
	'/bin',
	]),
	'PKG_CONFIG_PATH': '/usr/lib/arm-linux-gnueabihf/pkgconfig',
	})
	build_arm64 = build_arm + Environment({
	'BUILD_TARGET': 'aarch64-unknown-linux-gnu',
	'CC': 'aarch64-linux-gnu-gcc',
	'CXX': 'aarch64-linux-gnu-g++',
	'PATH': ':'.join([
	'{{ common.servo_home }}/bin',
	'/usr/local/sbin',
	'/usr/local/bin',
	'/usr/bin',
	'/usr/sbin',
	'/sbin',
	'/bin',
	]),
	'PKG_CONFIG_PATH': '/usr/lib/aarch64-linux-gnu/pkgconfig',
	'SERVO_RUSTC_WITH_GOLD': 'False',
	})

. That's why we see an issue when Cargo builds backtrace with a non-cross target. The mystery remains as to why that's happening - any suggestions for how to figure that out @alexchrichton?

[jdm]

Oh, I know why this is hitting us on servo/servo#18821 - the new version of metadeps uses error-chain 0.10, which depends on backtrace. dbus and x11-rs both have a build dependency on metadeps, which means that we now compile backtrace for the host system, leading to the situation where we end up selecting the wrong compiler.

[alexcrichton]

Interesting! Perhaps this is a case where the env var should be CC_arm-linux-gnueabihf-gcc instead of just CC?

[jdm]

That sounds like a good solution; thanks!