feat: configure cross to support x86_64 and aarch64 for linux/musl

[carlocorradini]

Related to PR #671 and PR #673

What has been changed:

• Configure cross for enhanced cross compilation supporting x86_64-unknown-linux-musl and aarch64-unknown-linux-musl and more...

[h3nill]

this is failing to build for me with error:

Error:
   0: failed to parse file `"/home/henil/Workspace/bytebeam/rumqtt/Cross.toml"` as TOML
   1: invalid type: map, expected a string for key `target.x86_64-unknown-linux-musl.image` at line 8 column 14

$ cross version
cross 0.2.5

[carlocorradini]

@henil You must follow the official installation procedure from Cross docs:

cargo install cross --git https://github.com/cross-rs/cross

I don't know why they want to install using git and not directly from cargo but they decided to do this :/

The CI respect their decision :)

[h3nill]

gotcha, works when installed via git and cargo.

btw why was gnu build removed, We would like to have both gnu and musl build if possible

[carlocorradini]

@henil The top level [build] is responsbile for a default implementation (gnu, etc...)
Since musl is particular the two targets only override the default one ([build]).
gnu is and always must be supported :)

[h3nill]

oh my bad missed that, been a while since i used cross :)

86_64-unknown-linux-gnu and x86_64-unknown-linux-musl are building properly 🎉

but aarch64-unknown-linux-musl fails to build for me (also tried disabling buildkit), i am running using:

cross build --target aarch64-unknown-linux-musl

with error (click to expand)

[+] Building 1.2s (5/5) FINISHED                                                                                                                 docker:default
 => [internal] load build definition from Dockerfile.aarch64-unknown-linux-musl-custom                                                                     0.0s
 => => transferring dockerfile: 212B                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/alpine:edge                                                                                             0.8s
 => CACHED [1/2] FROM docker.io/library/alpine:edge@sha256:2d01a16bab53a8405876cec4c27235d47455a7b72b75334c614f2fb0968b3f90                                0.0s
 => ERROR [2/2] RUN eval "apk add --no-cache build-base openssl-dev"                                                                                       0.3s
------
 > [2/2] RUN eval "apk add --no-cache build-base openssl-dev":
0.329 exec /bin/sh: exec format error
------
Dockerfile.aarch64-unknown-linux-musl-custom:5
--------------------
   3 |                     ARG CROSS_DEB_ARCH=
   4 |                     ARG CROSS_CMD
   5 | >>>                 RUN eval "${CROSS_CMD}"
--------------------
ERROR: failed to solve: process "/bin/sh -c eval \"${CROSS_CMD}\"" did not complete successfully: exit code: 1
Error:
   0: could not run container
   1: when building custom image
   2: when pre-building
   3: `docker buildx build --platform linux/arm64 --progress auto --label 'org.cross-rs.for-cross-target=aarch64-unknown-linux-musl' --label 'org.cross-rs.runs-with=aarch64-unknown-linux-musl' --label 'org.cross-rs.workspace_root=/home/henil/Workspace/bytebeam/rumqtt' --tag localhost/cross-rs/cross-custom-rumqtt:aarch64-unknown-linux-musl-d90ee-pre-build --build-arg 'CROSS_CMD=apk add --no-cache build-base openssl-dev' --build-arg 'CROSS_DEB_ARCH=musl-linux-arm64' --file /home/henil/Workspace/bytebeam/rumqtt/target/aarch64-unknown-linux-musl/Dockerfile.aarch64-unknown-linux-musl-custom --output 'type=docker' /home/henil/Workspace/bytebeam/rumqtt` failed with exit status: 1

Location:
   /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/convert/mod.rs:717

Warning: call to docker failed
Suggestion: is `buildx` available for the container engine?
Note: disable the `buildkit` dependency optionally with `CROSS_CONTAINER_ENGINE_NO_BUILDKIT=1`
Note: CROSS_CMD=apk add --no-cache build-base openssl-dev

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

How are you running it for aarch target?

[carlocorradini]

@henil Try starting buildx:

docker buildx create --name mybuilder --use --bootstrap

See this for more information

Moreover, try update Docker (I don't have found anything about buildx in Cross docs)

[carlocorradini]

@henil Any update? 🤗🥳

[h3nill]

i was working on this only :)

turns out to build for aarch target you need to install qemu and one other related package, see: https://stackoverflow.com/questions/71531771/docker-buildx-armv7-platform-missing. can you mention this somewhere? (also due to emulation compilation is very slow)

PS: sorry for the delay it was kinda hard to notice that i was missing arm platform support in docker so took a while to debug.

[carlocorradini]

I don't have qemu in my machine and it works 😅

[h3nill]

what machine are you on?

[carlocorradini]

@henil See https://www.docker.com/blog/how-to-rapidly-build-multi-architecture-images-with-buildx

I have already configured Docker buildx. I think this is the source of errors. Start/Configure Docker buildx with:

docker buildx create --name mybuilder --use --bootstrap

Restart Docker daemon and rebuild with cross

cross use qemu only for test (https://github.com/cross-rs/cross#supported-targets)

[carlocorradini]

what machine are you on?

WSL2 on Windows -> If it works on WLS2 it should/must work on a "true" Linux machine :)

[h3nill]

i have buildx image running already, before installing qemu it didn't show arm platforms here.

[carlocorradini]

Well install qemu and setup Docker buildx also in CI

[carlocorradini]

I've created the script for building the binaries (see PR #673)
I can add a check for qemu?

[carlocorradini]

Nevertheless this is strange because from cross docs qemu is used only for tests

[h3nill]

can anyone else with linux system try this?

cc: @de-sh @swanandx

[h3nill]

@carlocorradini I confirmed with someone else as well, aarch binaries doesn't build unless you have qemu related dependencies installed (i think its probably a docker thing not a cross), can you mention this two dependencies qemu qemu-user-static inside Cross.tom in comments.

ref: https://stackoverflow.com/questions/71531771/docker-buildx-armv7-platform-missing

[carlocorradini]

Good to know!

[de-sh]

i think its probably a docker thing not a cross

It's a cross thing, as in qemu is used to emulate hardware for running/testing against expected backend.

[carlocorradini]

@henil Done

[carlocorradini]

In the script available in PR #673 I'll add a check before running cross to assert the presence of qemu and qemu-user-static

[h3nill]

as in qemu is used to emulate hardware for running/testing against expected backend

@de-sh but running the following command shouldn't run binary or tests

cross build --target aarch64-unknown-linux-musl

[h3nill]

Thank you @carlocorradini! 🎉

[carlocorradini]

@henil build.sh script in PR #673 has been updated to check if qemu and qemu-user-static are installed