Support a convention for specifying more paths to toolchain binaries

[luser]

Currently cargo supports setting linker and ar in a config file, which is useful but is not always sufficient for cross-compiling crates that build native libraries.

While cross-compiling Servo I found that I had to specify quite a few extra tools to get everything to compile:
https://gist.github.com/luser/a33e5070d1c55a7d2c46fe763a9d1543#file-servo-cross-mac-L59

This was mostly because various crates used different build systems for building native libraries--autotools, cmake, etc. I think it'd be useful if the gcc crate supported a convention for specifying more tools in .cargo/config so that common crates for building native libraries (like cmake) could use that support to drive their builds.

Specifically, I think it would be useful if it allowed at least setting cc, cxx, ranlib, and exposed those values through its API.

[luser]

Interestingly, AFAICT cargo does not expose the values of linker or ar from .cargo/config to build scripts even if they're set.

[alexcrichton]

Yeah currently Cargo doesn't expose linker and/or ar to build scripts, but that probably wouldn't push this over the finish line either as you probably also want to at least read cxx (if not cc before linker). At the very least it'd be good to canonicalize this information all into one location, however, even if Cargo isn't passing the information.

It's probably worth at least doing some parsing of .cargo/config to learn information like this. I'd need to double check to ensure a build script can figure out what directory it was invoked from, however. I forget if that's passed through or not...

[GlenDC]

I'd need to double check to ensure a build script can figure out what directory it was invoked from

That depends on the user. I think you can specify the linker just by giving the name, in case it is in your environment PATH. Personally I don't have it in the path and just gave the full (absolute) location to the file, which would be enough to get the directory, as all the different tools can find in the same directory as the linker.

But even if the user doesn't give the full path, we should be able to get the directory of the executable on all development OS's, no? I know on Unix based systems it's possible with tools like whereis/which, and I suppose Windows has its own system call to get it as well.

[jdub]

I just noticed that gcc doesn't take advantage of the standard-ish TOOLPREFIX or CROSS_COMPILE environment variables in HOST != TARGET scenarios. Would you accept a change to use these when available?

[alexcrichton]

@jdub certainly! I hadn't heard of those before but if they're commonly in use I'd love to have support for them.

[jdub]

I might stick to CROSS_COMPILE for now, as it's unclear if TOOLPREFIX and TOOLCHAIN_PREFIX are sufficiently standard-ish. CROSS_COMPILE appears in Linux, musl, etc. as the binary name plus a hyphen, such that you can use it and get a valid binary name even if it's not set, e.g.

CC := $(CCACHE) $(CROSS_COMPILE)gcc
CXX := $(CCACHE) $(CROSS_COMPILE)g++
AR := $(CROSS_COMPILE)ar

[alexcrichton]

Ok sounds good to me!

[pietro]

FYI, cargo exposes the linker configuration in the RUSTC_LINKER environment variable.

[21oavery]

It looks like cargo will pass the environment variable TARGET to build scripts

https://doc.rust-lang.org/cargo/reference/environment-variables.html

Also of note are HOST and NUM_JOBS

[21oavery]

And also RUSTC_LINKER, which does what it says

[samliddicott]

Could we also have TARGET_LD and LD distinguished?

I'm cross-compiling from one x86_64-unknown-linux-gnu to a different x86_64-unknown-linux-gnu

TARGET_CC and CC might work for compiling, to distinguish between build scripts and the cross-compiled binaries/libs, but the same linker is being used to build the build scripts as is being used to link the cross-compiled binaries, and so the build scripts won't run.

I've had to set this script as my: --config target.$(RUST_TARGET).linker=\"linker-hack\" in order to call the right linker, depending on what is being linked

#! /bin/bash
grep -q 'build_script_main\|build_script_build' <<<"$*" && exec gcc "$@" || exec ${TARGET_CC:-${CC}} "$@"

[luser]

I'm cross-compiling from one x86_64-unknown-linux-gnu to a different x86_64-unknown-linux-gnu

Can you expound on how the two targets differ? Also: I don't think this is the right place to ask this question. This is about the cc crate, but it sounds like your issue is with cargo itself choosing the right linker. You may be interested in one or both of these cargo issues that I filed a while back:
rust-lang/cargo#5754
rust-lang/cargo#5755

[samliddicott]

Can you expound on how the two targets differ?

Different compilers, different libc, and ultimately to run against a different kernel. target binaries fail to run in the host environment due to linker problems.

Those tickets you referred to also seem relevant, I'm not sure which (or both?) would need adjusting.

[samliddicott]

I think that these reports all cover my use case thoroughly:

• Support for build scripts where host and target architecture match rust-embedded/meta-rust-bin#97
• choice of TARGET_CC or HOST_CC does not seem consistent cargo#8147
• Provide a way to specify configuration for host compilation cargo#5754
• Indicate to build scripts that the crate they are being run for is being built for the host cargo#5755

but I think this one really is the point:

• Add the ability to provide build flags for the build-script-build cargo#3349

[jameshilliard]

TARGET_CC and CC might work for compiling, to distinguish between build scripts and the cross-compiled binaries/libs, but the same linker is being used to build the build scripts as is being used to link the cross-compiled binaries, and so the build scripts won't run.

Maybe the target-applies-to-host and/or host-config feature is what is needed here?

You should be able to override the host linker explicitly when using stable toolchains by setting this in your env:

__CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly"
CARGO_UNSTABLE_TARGET_APPLIES_TO_HOST="true"
CARGO_TARGET_APPLIES_TO_HOST="false"
CARGO_UNSTABLE_HOST_CONFIG="true"
CARGO_HOST_LINKER="host-gcc"

Setting CARGO_UNSTABLE_HOST_CONFIG with the explicit CARGO_HOST_LINKER might not actually be needed as cargo should by default use the normal host linker as long as the __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS, CARGO_UNSTABLE_TARGET_APPLIES_TO_HOST and CARGO_TARGET_APPLIES_TO_HOST are set as in my example

[samliddicott]

Maybe the target-applies-to-host and/or host-config feature is what is needed here?

Thank you so much @jameshilliard, these touch the point so well, particularly host-config, and the tip to also apply to current stable is very gratefully received.

I am surprised that these don't both have the same value, is that a mistake? :

CARGO_UNSTABLE_TARGET_APPLIES_TO_HOST="true"
CARGO_TARGET_APPLIES_TO_HOST="false"

For stable, if I set the env as described, do I override the host linker explicitly by setting HOST_LD or HOST_CC, or by using a host.* section in config.toml as explained in host-config ?

EDIT: I see your example here - thanks: https://github.com/buildroot/buildroot/blob/2022.11-rc1/package/pkg-cargo.mk#L26-L47

[jameshilliard]

I am surprised that these don't both have the same value, is that a mistake? :

Should be correct, see:
CARGO_UNSTABLE_TARGET_APPLIES_TO_HOST="true"
CARGO_TARGET_APPLIES_TO_HOST="false"

For stable, if I set the env as described, do I override the host linker explicitly by setting HOST_LD or HOST_CC, or by using a host.* section in config.toml as explained in host-config ?

You can set this if you need a specific host linker:

CARGO_UNSTABLE_HOST_CONFIG="true"
CARGO_HOST_LINKER="host-gcc"