cross-compilation support

[Gankra]

This is punted on in the current version, pending some evaluation of the options. There are three four options I know:

• cross-rs (established, based on docker)
• cargo-zigbuild (new kid on the block, has some blocking issues)
• cargo-xwin (very specific to targetting windows? just learned about this.)
• Do It Our Dang Selves With Rustup (please god no)

This is a blocker for musl and aarch64 (arm64) support.

Note that taiki-e/upload-rust-binary-action does handle this and in my experiments whatever it does Just Works (great work from taiki-e as always).

[Gankra]

One of my big concerns about cross-compilation is to what extent certain Fancy Things cannot be cross-compiled.

Apple certainly does not want you cross-compiling from linux to macos. I'm not sure if support for code signing (#21) or certain installers is necessarily non-crossable. Certainly some are harder to cross. Certainly there are... Ways Around The Rules. But I don't want to get people in trouble or have them ship busted things!

[reubenmiller]

Just some of my recent experience/pain regarding cross compiling for Linux arm* and x86_64 targets for an open source project that I'm involved with.

Basically we have to build for linux environments on arm >= v6, arm64 and x86_64, for both gnu and musl releases. We were using cargo-cross however still had some issues to build for all the targets and the output was dependant on your build machine.

Then I saw a great blog post about cargo-zigbuild and was amazed with how effort it was to cross compile for all of our targets, especially since the other developers have a mix between Windows (using WSL2) and MacOS (M1/arm64 and amd64) setups.

Below details some of the details with the projects:

Experience cargo-cross

• Slow builds as building on MacOS M1 (arm64/aarch64) and the current default is to use amd64 images by default (though there are unofficial arm64 available...there is also a "zigbuild" variant, from what I found digging around)
• Building for musl on some targets just didn't work (mostly because of our project's dependency to the common ring...the number of times that dependency has caused me build headaches is just astounding)

Experience with cargo-zigbuild

• Builds are quick
• Does not rely on docker
• Feels more like rust, as you just use cargo target add xxx, then cargo zigbuild --target xxx
• Dependencies are included (ok, still had to install ziglang via pip first but that was it)

Overall I was finally able to get the following targets built using cargo-zigbuild running both natively on MacOS M1 and inside a linux (dev) container.

• x86_64-unknown-linux-musl
• aarch64-unknown-linux-musl
• armv7-unknown-linux-musleabihf
• arm-unknown-linux-musleabihf
• x86_64-unknown-linux-gnu
• aarch64-unknown-linux-gnu
• armv7-unknown-linux-gnueabihf (yes, this does actually work...I think notice in the project's README.md is out of date)
• arm-unknown-linux-gnueabihf

Though like you said, I don't think it will work for absolutely everything, so it might be worthwhile allowing some "composability" when it comes to building for different targets. And the targets are above are not so exotic compared to the other more lower level embedded targets, e.g. risc etc.

I created a gist to detail my experience, including the references to sources etc.

I hope my experience can at least help someone else, as this had caused me countless hours fluffy around with various dependencies/build machines.

[messense]

• armv7-unknown-linux-gnueabihf (yes, this does actually work...I think notice in the project's README.md is out of date)

To clarify, I have implemented some workarounds for these caveats in cargo-zigbuild and updated the readme: https://github.com/rust-cross/cargo-zigbuild#caveats

[VorpalBlade]

Is there any work being done on this and/or is there any way to help on this? Cross compiling to Linux ARM and Linux Aarch64 is an absolute must for me.

[VorpalBlade]

@reubenmiller I found the opposite. Zigbuild is great when it works, but:

1. It fails when trying to for example cross compile and link OpenSSL (openssl-sys crate). You can either enabled the vendored feature of openssl-sys to build OpenSSL from source or with cross install the relevant -dev package for cross compiling in a pre-build step in the docker container.
   Installing OpenSSL as a pre-build step is much faster than building it from scratch every time, easily negating the general speed improvement of zigbuild over cross. (And no, rustls/ring is not an option for several of the platforms I'm targeting.)
2. Cross supports running the tests using user mode qemu, zigbuild doesn't as far as I have been able to find. While this is not important for release, it is important for CI in general. And I'd rather not have to maintain two separate cross compilation systems.
3. Cross compiling to Windows (x86_64-pc-windows-gnu) from Linux just doesn't seem to work at at all with zigbuild, works out of the box with cross.

Thus I think cross is the better and more mature option to go with.

[cryingpotat0]

👋 Hey! I also wanted to chime in and see if cross compilation for cost reduction purposes was on the roadmap? Spinning up separate machines (especially expensive Mac machines, almost 10x an equivalent Linux machine) for every release seems like it could blow past the free tier on Github actions. While it's still cheap in the grand scheme of things, it would be cool to use the cheapest Github actions instance to compile for all platforms where possible.

[ssokolow]

Unfortunately, mac machines are probably the most difficult to replace because, unlike with MinGW, there's no libre equivalent to some of the parts of Apple's toolchain. (I believe the binary-signing tool is one such example, but that may be outdated information.)

The solutions I've run into which work without mac hardware have involved using parts of Apple's toolchain on top of non-mac hardware in violation of its license.

[frol]

I hope this issue can be prioritized. It seems that a number of projects would love to see it happen:

• chore: set up cargo-dist kaspar030/laze#303
• Please support more platforms for the pre-built binaries probe-rs/probe-rs#1892
• Build linux-musl targets stchris/aleph-tui#3
• I have 3+ CLIs that would also benefit from it:
  • https://github.com/near/near-cli-rs
  • https://github.com/near/cargo-near
  • https://github.com/bos-cli-rs/bos-cli-rs

[Toasterson]

For MacOS there are decent hosted runner options available in gitlab and github it's only windows that due to DRM is hard to get. But there cross compiling works very well with ubuntu runners and native packages and the gnome msitools. I opened #821 for integration of the later.

[VorpalBlade]

For MacOS there are decent hosted runner options available in gitlab and github it's only windows that due to DRM is hard to get. But there cross compiling works very well with ubuntu runners and native packages and the gnome msitools. I opened #821 for integration of the later.

No I don't believe this is true. Github Actions offers free (for open source) Ubuntu, Windows and MacOS x86-64 runners. There are no free MacOS ARM64 runners. Thus you need cross compilation if you want to create native MacOS-ARM64 binaries (cross testing seems impossible, rosetta only goes the other direction as far as I understand).

As far as I know most non-github options cost money, which is not an option for small open source hobby projects.

Windows can also be trivially cross compiled from Linux with the -gnu toolchain. The -msvc one cannot. There are no license-abiding ways to cross compile to MacOS at all as far as I know.

[Toasterson]

Hi, @VorpalBlade I was talking about Gitlab exclusively as we cannot use GitHub due to corporate requirements. GitLab has ARM64 runners. https://docs.gitlab.com/ee/ci/runners/saas/macos_saas_runner.html#machine-types-available-for-macos

[VorpalBlade]

Hi, @VorpalBlade I was talking about Gitlab exclusively as we cannot use GitHub due to corporate requirements. GitLab has ARM64 runners. https://docs.gitlab.com/ee/ci/runners/saas/macos_saas_runner.html#machine-types-available-for-macos

You said "gitlab and github" in your original text so that is what I responded to. However it seems that the MacOS runners on gitlab are only available on "Tier: Premium, Ultimate", so that is irrelevant for open source projects. At least the windows runners there are available on the free tier.

[Toasterson]

Oh, sorry. I noticed that now. Anyway, in the text I was looking at it said

SaaS runners on macOS are in Beta for open-source programs and customers in Premium and Ultimate plans.

So I took it for a spin on my free account. https://gitlab.com/Toasterson/rust-cross-testing/-/pipelines but it looks like this feature does indeed need a Subscription...

[connec]

I've been able to cross compile aarch64 from standard x86 GitHub Actions runners with a fairly small amount of configuration:

• Manually add the relevant targets to Cargo.toml (this is necessary as they're not offered as options by cargo dist init).

  E.g.

  # Target platforms to build apps for (Rust target-triple syntax)
  targets = ["aarch64-apple-darwin", "aarch64-unknown-linux-gnu", "aarch64-unknown-linux-musl", "x86_64-apple-darwin", "x86_64-unknown-linux-gnu", "x86_64-unknown-linux-musl"]

• Add gcc-aarch64-linux-gnu to cargo-dist's dependencies.apt table. This installs a cross-compilation toolchain which can be invoked as aarch64-linux-gnu-gcc.

  E.g.

  [workspace.metadata.dist.dependencies.apt]
  gcc-aarch64-linux-gnu = { version = '*', targets = ["aarch64-unknown-linux-gnu", "aarch64-unknown-linux-musl"] }

• Create a .config/cargo.toml with the following overrides. cargo will automagically use aarch64-linux-gnu-gcc as the cc for aarch64-unknown-linux-gnu, but needs to be told to use it also for -musl and for linking of both targets.

  E.g.

  [env]
  CC_aarch64_unknown_linux_musl = "aarch64-linux-gnu-gcc"
  
  [target.aarch64-unknown-linux-gnu]
  linker = "aarch64-linux-gnu-gcc"
  
  [target.aarch64-unknown-linux-musl]
  linker = "aarch64-linux-gnu-gcc"

YMMV – I have minimal system dependencies in the projects where I've used this and I'm sure it would require more fiddling if you depend on additional system libraries (openssl etc). I've also not thoroughly tested the results (though I've tested that they execute, at least).

[frol]

@connec Thanks for sharing! I tried to follow your steps, but system dependencies are blocking me (openssl and libudev): near/near-cli-rs#319

[Toasterson]

Then you will need a sysroot with copies of those libs and headers and use the --sysroot flag when compiling. Adding that flag to the .cargo/config.toml CC lines worked for me when I needed to cross compile. Checkout the cross project for sysroot tarballs and OCI images in the wild.

[frol]

Well, I used cross before and it worked great, so I would love to see cargo-dist support it or any other solution.

I am getting more and more reports these days from people trying to use my CLIs on:

1. Docker on macOS ARM64 (M1/M2/...) which runs Linux ARM64 images
2. Raspberry Pi

@Gankra What is your vision/priority regarding cross-compilation? As a temporary solution, I am thinking about creating a separate workflow to build extra binary releases (I'll play with https://github.com/taiki-e/upload-rust-binary-action or build completely from scratch), and I expect that the installer will see those binary-release archives

[bjeanes]

YMMV – I have minimal system dependencies in the projects where I've used this and I'm sure it would require more fiddling if you depend on additional system libraries (openssl etc). I've also not thoroughly tested the results (though I've tested that they execute, at least).

It compiled every target except aarch64-unknown-linux-gnu, because of libudev:

error: failed to run custom build command for `libudev-sys v0.1.4`

Caused by:
  process didn't exit successfully: `/home/runner/work/modbus-mqtt/modbus-mqtt/target/dist/build/libudev-sys-289bc33b2b5e4d83/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=LIBUDEV_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_ALLOW_CROSS
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS
  cargo:rerun-if-env-changed=PKG_CONFIG_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libudev-sys-0.1.4/build.rs:38:41:
  called `Result::unwrap()` on an `Err` value: "pkg-config has not been configured to support cross-compilation.\n\nInstall a sysroot for the target platform and configure it via\nPKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_PATH, or install a\ncross-compiling wrapper for pkg-config and set it via\nPKG_CONFIG environment variable."
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

Based on reading the pkg-config-rs readme, in theory this could be overcome by setting the following in .cargo/config.toml:

[env]
PKG_CONFIG_SYSROOT_DIR_aarch64_unknown_linux_gnu = "/usr/aarch64-linux-gnu"

However, I'm not really sure how one goes about building a sysroot, nor how one would do this in a way that was harmonious with cargo-dist. I did find https://github.com/windelbouwman/sysroot-creator which looks interesting and could possibly be used by setting a wrapper script for the broken target which creates the sysroot with needed deps and exports the right PKG_CONFIG_* variables, but it does seem like an experiment from years ago that hasn't seen much usage or attention since.

Fortunately, in my case, libudev is a transitive dependency of serialport and I can live without it -- for now --, so can use serialport = { ..., default-features = false } to unblock some cross-compilation.

TL;DR: cross compilation out of the box seems advantageous. cargo-dist has a lot of secondary upsides in how it works, but https://github.com/taiki-e/upload-rust-binary-action supports cross-compilation and might be something to try for those who require cross-compilation beyond what can be worked around today here.

[Toasterson]

A Sysroot is simply a Target OS/Arch version of your Root filesystem layout. Simplest trick is to expand a VM image (Say ubuntu aarch64) into a directory and point pkgconf to it. Sysroot is at the end of the day a LLVM/GCC Linker feature all other utilities pass the sysroot to the linker.

[andrewdavidmackenzie]

cross not on your list?

From my experience (responding to comment about cross compiling directly from x86 host), that can work in many cases - but then you come across a crate that is a wrapper to a complex C/C++ build, and you run into header file and sysroot problems and it becomes very difficult.

So far, we have only have real (relatively straightforward) success building for aarch64, and arm7 using cross.

cargo-zigbuild looks amazing but we ran into a few blocker issues that prevented us from moving to it, so we stuck with cross and are happy.

IMO, no need to cross-compile macOS, if GH Actions has macOS runners (x86 and aarch64).