Cross compiling with sysroot

[andber1]

I have an armv7-unknown-linux-gnueabihf target with a rather old glibc 2.20. So the standard way of cross compiling does not work for me, because the glibc versions provided by the ghcr.io/cross-rs/armv7-unknown-linux-gnueabihf images do not match my target system. So my naive idea was, why not link against the target system itself? I have an image of the target system that I have mounted into the docker image so that I can tell the linker to use it as sysroot. I am not an expert when it comes to cross compiling, so maybe this idea does not work at all. It would be nice, if someone can help me.

These are my attempts so far:

Sysroot link attempts

I added these lines to Cross.toml in order to mount my sysroot and configuring the linker to use it:

[build]
default-target = "armv7-unknown-linux-gnueabihf"
[build.env]
volumes = ["MY_SYSROOT=/path/to/my/sysroot"]
passthrough = ["CARGO_TARGET_ARMV7_UNKNOWN_LINUX_GNUEABIHF_RUSTFLAGS=-C link-arg=--sysroot=/path/to/my/sysroot"]

Unfortunately I cannot use the environment variable MY_SYSROOT in the last line, because it would no be expanded. Off-topic question: Is there a simple solution for that?
I noticed that the linker does not use the libc from /path/to/my/sysroot/lib/libc-2.20.so (it linked without errors, but the resulting executable has GLIBC links greater 2.20 when examining with readelf). So I explicitly tell the linker to use that version by adding this line in build.rs:

println!("cargo:rustc-link-lib=c-2.20");

I asume that the linker uses libc from my sysroot now, but it still uses a different libpthread that is not compatible. For example I get the linker error:

/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/lib/../lib/libpthread.so: undefined reference to `_IO_enable_locks@GLIBC_PRIVATE'

What is the correct way to tell the linker to only look in my sysroot? Apparently it still looks in /usr/lib/gcc-cross/arm-linux-gnueabihf.
I also tried to specify the search paths in build.rs in addition.

println!("cargo:rustc-link-search=/path/to/my/sysroot/lib/");
println!("cargo:rustc-link-search=/path/to/my/sysroot/usr/lib/");

Then I get:

arm-linux-gnueabihf/bin/ld: cannot find /usr/lib/libpthread_nonshared.a inside /path/to/my/sysroot

libpthread_nonshared.a is not available on my target system. I am not sure why this happens. Can anyone explain this?

Creating new image with correct libc version

I tried to create a new docker image that has the correct libc version. I followed the instructions in the cross-toolchains repository and in this discussion. I did manage to create a new image using these commands:

xtask configure-crosstool armv7-unknown-linux-gnueabihf-cross --glibc-version 2.20 --gcc-version 4.9.1 --linux-version 3.0.35
xtask build-docker-image armv7-unknown-linux-gnueabihf-cross

In the end there was one warning. I am not sure if that can be ignored:

[Warning] one or more build args were not consumed: [CROSS_TARGET_TRIPLE]
Successfully tagged ghcr.io/cross-rs/armv7-unknown-linux-gnueabihf-cross:local

I was able to successfully compile a hello world program (without setting a sysroot for now). But it was using a new libc. And looking into the image I found this file: /usr/lib32/libc-2.31.so. Shouldn't that be 2.20?

Using zig

Now I tried to specifiy the correct libc version with zig. I added the following to my Cross.toml:

[build]           
zig = "2.20"

The sysroot was still set. The program compiled and linked successfully. But I get a core dump when running the executable on the target:

Core was generated by `./hello-world'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x4015063c in ?? () from /lib/ld-linux-armhf.so.3
warning: Unsupported auto-load script at offset 0 in section .debug_gdb_scripts
of file hello-world.

Does anyone have an idea what went wrong?

Using musl target

Using the target armv7-unknown-linux-musleabihf works out of the box. However, I am curious why my attempts above do not work. Maybe someone has an idea?

[Emilgardis]

I was able to successfully compile a hello world program (without setting a sysroot for now). But it was using a new libc. And looking into the image I found this file: /usr/lib32/libc-2.31.so. Shouldn't that be 2.20?

No, the sysroot is elsewhere (/x-tools/), not in /usr/

I'm not sure what went wrong with zig, as for the attempt to borrow the sysroot from the target, not going to delve into it :3

[andber1]

Thanks for the clarification. But even the libc version in x-tools is not 2.20 as I have specified (and the hello-world executable also uses the wrong libc).

$ /x-tools/armv7-unknown-linux-gnueabihf/armv7-unknown-linux-gnueabihf/sysroot/usr/bin/ldd --version
ldd (crosstool-NG UNKNOWN) 2.38

Do you have any idea what went wrong when creating the new image? I have used this file:

crosstool-ng/armv7-unknown-linux-gnueabihf-cross.config.in

CT_CONFIG_VERSION="4"
CT_PREFIX_DIR="/x-tools/${CT_TARGET}"
CT_DOWNLOAD_AGENT_CURL=y
CT_ARCH_ARM=y
CT_ARCH_ARM_MODE_THUMB=y
CT_ARCH_SUFFIX="v7"
CT_ARCH_ARCH="armv7-a"
CT_ARCH_FPU="vfpv3-d16"
CT_ARCH_FLOAT_HW=y
CT_KERNEL_LINUX=y
%CT_LINUX_V%
%CT_LINUX%
CT_BINUTILS_V_2_32=y
%CT_GLIBC_V%
%CT_GLIBC%
%CT_GCC_V%
%CT_GCC%
CT_CC_LANG_CXX=y
CT_GETTEXT_V_0_19_8_1=y
CT_GMP_V_6_1=y
CT_ISL_V_0_20=y
CT_LIBICONV_V_1_15=y
CT_NCURSES_V_6_1=y

After this command

xtask configure-crosstool armv7-unknown-linux-gnueabihf-cross --glibc-version 2.20 --gcc-version 4.9.1 --linux-version 3.0.35

this file was created:

crosstool-config/armv7-unknown-linux-gnueabihf-cross.config

CT_CONFIG_VERSION="4"
CT_PREFIX_DIR="/x-tools/${CT_TARGET}"
CT_DOWNLOAD_AGENT_CURL=y
CT_ARCH_ARM=y
CT_ARCH_ARM_MODE_THUMB=y
CT_ARCH_SUFFIX="v7"
CT_ARCH_ARCH="armv7-a"
CT_ARCH_FPU="vfpv3-d16"
CT_ARCH_FLOAT_HW=y
CT_KERNEL_LINUX=y
CT_LINUX_V_3_0=y
# CT_LINUX_NO_VERSIONS is not set
CT_LINUX_VERSION="3.0.35"
CT_LINUX_older_than_4_8=y
CT_LINUX_4_8_or_older=y
CT_LINUX_older_than_3_7=y
CT_LINUX_3_7_or_older=y
CT_LINUX_older_than_3_2=y
CT_LINUX_3_2_or_older=y
CT_BINUTILS_V_2_32=y
CT_GLIBC_V_2_20=y
# CT_GLIBC_NO_VERSIONS is not set
CT_GLIBC_VERSION="2.20"
CT_GLIBC_2_29_or_older=y
CT_GLIBC_older_than_2_29=y
CT_GLIBC_2_27_or_older=y
CT_GLIBC_older_than_2_27=y
CT_GLIBC_2_26_or_older=y
CT_GLIBC_older_than_2_26=y
CT_GLIBC_2_25_or_older=y
CT_GLIBC_older_than_2_25=y
CT_GLIBC_2_24_or_older=y
CT_GLIBC_older_than_2_24=y
CT_GLIBC_2_23_or_older=y
CT_GLIBC_older_than_2_23=y
CT_GLIBC_2_20_or_older=y
CT_GLIBC_2_17_or_later=y
CT_GLIBC_later_than_2_14=y
CT_GLIBC_2_14_or_later=y
CT_GCC_V_4=y
# CT_GCC_NO_VERSIONS is not set
CT_GCC_VERSION="4.9.1"
CT_CC_LANG_CXX=y

CT_GETTEXT_V_0_19_8_1=y

CT_GMP_V_6_1=y

CT_ISL_V_0_20=y

CT_LIBICONV_V_1_15=y

CT_NCURSES_V_6_1=y

After

xtask build-docker-image armv7-unknown-linux-gnueabihf-cross

I got a new image. So basically I followed the instructions in this discussion as already mentioned.
But the libc version is wrong. Do you see the mistake?

Regarding the sysroot issue: I can understand if you do not want to delve into it. However, can you tell me if you see a general flaw in my idea? It should be possible to link against libc from my sysroot, right?

[Emilgardis]

ldd --version only shows what was linked when it was built, you'll need to check the .so
strings /x-tools/armv7-unknown-linux-gnueabihf/armv7-unknown-linux-gnueabihf/sysroot/usr/lib/libc.so | grep "GLIBC"

Yes, theorethically what you did with copying could work, what you would have to do is undo the config we've done with env-vars

[polarathene]

I don't use cross myself, but have plenty of experience with cross-compilation via cargo-zigbuild with sysroots managed via Fedora dnf's --installroot + --forcearch options, then updating some ENV to reference those when building.

Zig makes targeting lower glibc (or specific release) pretty straight-forward, but when you should also keep in mind situations where your build will link to libs in it's search path, and that differs from runtime resolving of deps.

For example I can build for aarch64 on an x86_64 host with a custom sysroot that works well. However when I try to inspect the deps on that same system with ldd (more specifically the aarch64 version in the sysroot)

• It will now use runtime search paths, where /etc/ld.so.cache + LD_LIBRARY_PATH are relevant.
• As the binary likely only has the library name to resolve, in this scenario it will attempt to resolve from the x86_64 root, not the sysroot as it knows nothing about that (and really shouldn't for distribution).
• If I take that binary and use ldd from within a chroot of that sysroot, or on a aarch64 host, then it'll resolve to the correct library paths.

In some cases you may want to link to libraries that are not part of the host system, but bundled with your binary for distribution. These can be linked to by having your binary built (or in post via tools like patchelf) configure RPATH / RUNPATH with $ORIGIN (not an ENV but special keyword string) as the base relative path from the binary, libraries will then resolve as expected for RPATH which will apply that prefix recursively, while RUNPATH will need to be applied to each individual library you bundle otherwise the transitive deps will fallback to resolving elsewhere instead as they'd no longer be away of the RUNPATH (RPATH is deprecated, but provides such functionality). LD_LIBRARY_PATH is used for resolution after RPATH and before RUNPATH, the behaviour also differs on Alpine / musl, where RPATH and RUNPATH are treated the same IIRC (effectively like LD_LIBRARY_PATH but encoded into the binary/library).

---

From other information you shared it sounds like you were trying to build static glibc? This is generally discouraged as it's not proper static like musl.

• In some cases you can use Eyra instead (requires nightly Rust toolchain and may not be compatible with some foreign source deps depending on the glibc symbols).
• You can control the external libs for a portable build without fully static linking glibc, by using RPATH/RUNPATH as mentioned. If your linked deps aren't that many, this is viable should you need to resort to this. It would be independent of the host system, so you could run such on Alpine for example as there should be no linking to any libraries provided by the system itself.
• Ideally with Zig it should work just fine if you lower the glibc target for compatibility, but make sure you are providing the correct sysroot for any header includes needed, along with explicit search path (eg: RUSTFLAGS='-L <your sysroot path>/usr/lib64') for that sysroot (Zig excludes these for cross-compilation and cargo-zigbuild also forces -nostdinc so adjusting sysroot alone probably won't be sufficient). You may need to additionally set other ENV when relevant like the PKG_CONFIG_SYSROOT_DIR or CFLAGS='-isystem <your sysroot path>/usr/include', etc.

Oh and Zig doesn't support static glibc, it will segfault in earlier releases and fail/error on the current 14 series IIRC, but it is being enabled as a supported option with Zig 15 but not for non-native/custom targets which includes targeting different glibc than system IIRC, cargo-zigbuild also needs to make a change to support this once available AFAIK.

[Emilgardis]

Thank you for sharing your expertise!