CI Cross-building

[konsumer]

This may be out of scope for this project, but I am having a heck of a time getting a rust SDL2 project to build in Github CI for linux/mac for arm64, and hope that someone has some experience with this.

Here is what I tried:

For my first pass, I just download SDL for each platform and build it normally. Here is a complete x86-64 build, that seems to work ok.

I tried running the outputted mac x86-64 build on an M1, and although it will start (I think because mac triggers an emulator on incorrect platform) it fails trying to link against my system-installed (via brew) SDL2.

dyld[17465]: Library not loaded: /usr/local/opt/sdl2/lib/libSDL2-2.0.0.dylib
  Referenced from: <874CD34A-B1B6-3194-94BD-85C4C8CE2FCE> /Users/konsumer/Desktop/rm8/rm8-mac-x86_64/rm8
  Reason: tried: '/Users/konsumer/VulkanSDK/1.3.216.0/macOS//lib/libSDL2-2.0.0.dylib' (no such file), '/libSDL2-2.0.0.dylib' (no such file), '/usr/local/opt/sdl2/lib/libSDL2-2.0.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/sdl2/lib/libSDL2-2.0.0.dylib' (no such file), '/usr/local/opt/sdl2/lib/libSDL2-2.0.0.dylib' (no such file), '/usr/local/lib/libSDL2-2.0.0.dylib' (no such file), '/usr/lib/libSDL2-2.0.0.dylib' (no such file, not in dyld cache)
[1]    17465 abort      ./rm8

If I understand the system correctly, I need to use lipo to make a universal binary for both platforms, in a single executable.

The next step is trying to cross-build for arm64 on mac/linux.

On linux, this fails:

error: failed to run custom build command for `libc v0.2.139`

Caused by:
  process didn't exit successfully: `/target/release/build/libc-553f84[96](https://github.com/konsumer/rm8/actions/runs/3783838112/jobs/6432688520#step:6:97)44afbfd2/build-script-build` (exit status: 1)
  --- stderr
  /target/release/build/libc-553f849644afbfd2/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /target/release/build/libc-553f849644afbfd2/build-script-build)
  /target/release/build/libc-553f849644afbfd2/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /target/release/build/libc-553f849644afbfd2/build-script-build)
  /target/release/build/libc-553f849644afbfd2/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /target/release/build/libc-553f849644afbfd2/build-script-build)

Which seems like an issue with cross, but I can't really tell.

Mac has a similar fail:

error[E0463]: can't find crate for `core`
  |
  = note: the `aarch64-apple-darwin` target may not be installed
  = help: consider downloading the target with `rustup target add aarch64-apple-darwin`

error[E0463]: can't find crate for `compiler_builtins`

error[E0463]: can't find crate for `core`
   --> /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.3.2/src/lib.rs:282:1
    |
282 | pub extern crate core as _core;
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate
    |
    = note: the `aarch64-apple-darwin` target may not be installed
    = help: consider downloading the target with `rustup target add aarch64-apple-darwin`

I think this may be related to cross not correctly installing the targets it needs, so I add rustup target add aarch64-unknown-linux-gnu and rustup target add aarch64-apple-darwin to linux/mac. That fails the same on linux and has an SDL2 linkage error on Mac.

This may be an entirely different issue, but on my M1, I tried to simulate building (in docker) the other way (linux arm64 building for linux x86-64) without cross, and that didn't work either (I get a pkg-config error):

# uname -a
Linux 46b739d74d77 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021 aarch64 GNU/Linux

# apt install -y libsdl2-dev libudev-dev
libsdl2-dev is already the newest version (2.0.14+dfsg2-3+deb11u1).
libudev-dev is already the newest version (247.3-7+deb11u1).

# rustup target add x86_64-unknown-linux-gnu
info: downloading component 'rust-std' for 'x86_64-unknown-linux-gnu'
info: installing component 'rust-std' for 'x86_64-unknown-linux-gnu'
 29.7 MiB /  29.7 MiB (100 %)  19.1 MiB/s in  1s ETA:  0s

# cargo build --release --target x86_64-unknown-linux-gnu
    Updating crates.io index
  Downloaded bitflags v1.3.2
  Downloaded proc-macro2 v1.0.49
  Downloaded ryu v1.0.12
  Downloaded syn v1.0.107
  Downloaded unicode-ident v1.0.6
  Downloaded serialport v4.2.0
  Downloaded version-compare v0.1.1
  Downloaded static_assertions v1.1.0
  Downloaded serde_derive v1.0.151
  Downloaded serde v1.0.151
  Downloaded serde_json v1.0.91
  Downloaded nix v0.26.1
  Downloaded itoa v1.0.5
  Downloaded nix v0.24.3
  Downloaded lazy_static v1.4.0
  Downloaded libudev-sys v0.1.4
  Downloaded sdl2 v0.35.2
  Downloaded libc v0.2.139
  Downloaded pkg-config v0.3.26
  Downloaded libudev v0.3.0
  Downloaded ctrlc v3.2.4
  Downloaded cfg-if v1.0.0
  Downloaded quote v1.0.23
  Downloaded sdl2-sys v0.35.2
  Downloaded 24 crates (9.6 MB) in 0.92s (largest was `sdl2-sys` at 7.4 MB)
   Compiling libc v0.2.139
   Compiling bitflags v1.3.2
   Compiling cfg-if v1.0.0
   Compiling libudev-sys v0.1.4
error: failed to run custom build command for `libudev-sys v0.1.4`

Caused by:
  process didn't exit successfully: `/src/target/release/build/libudev-sys-ed6dac626792d036/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=LIBUDEV_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_ALLOW_CROSS
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "pkg-config has not been configured to support cross-compilation.\n\nInstall a sysroot for the target platform and configure it via\nPKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_PATH, or install a\ncross-compiling wrapper for pkg-config and set it via\nPKG_CONFIG environment variable."', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/libudev-sys-0.1.4/build.rs:38:41
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish..

Does anyone know any tricks to getting this to work?

[konsumer]

I managed to cross-build linux arm64 in docker/qemu, which is maybe not the "rust way" and is slow, but seems to work.

So, I guess the only thing left is mac arm64. Should I try to just build it for x86_64, but static? I think this would trigger the emulator on M1, and since SDL2 would be inlined, it would use that, instead of system SDL2 with arch-mismatch. What is the best way to do this? I tried adding bundled and static-link features, but that didn't work for me:

error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-arch" "x86_64" "-m64" "/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/rustcJJpEQG/symbols.o" "/Users/runner/work/rm8/rm8/target/release/deps/rm8-0bad5a46bbd9eba9.rm8.57df0b02-cgu.5.rcgu.o" "-L" "/Users/runner/work/rm8/rm8/target/release/deps" "-L" "/Users/runner/work/rm8/rm8/target/release/build/sdl2-sys-254e8e6752391830/out/lib" "-L" "/Users/runner/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/rustcJJpEQG/libsdl2_sys-358bf0a65f35dd6c.rlib" "/Users/runner/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-bda477245996038f.rlib" "-framework" "IOKit" "-framework" "CoreFoundation" "-framework" "Cocoa" "-framework" "IOKit" "-framework" "Carbon" "-framework" "ForceFeedback" "-framework" "GameController" "-framework" "CoreVideo" "-framework" "CoreAudio" "-framework" "AudioToolbox" "-framework" "Metal" "-liconv" "-lSystem" "-lc" "-lm" "-L" "/Users/runner/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "-o" "/Users/runner/work/rm8/rm8/target/release/deps/rm8-0bad5a46bbd9eba9" "-Wl,-dead_strip" "-nodefaultlibs"
  = note: Undefined symbols for architecture x86_64:
            "_CHHapticDynamicParameterIDHapticIntensityControl", referenced from:
                -[SDL_RumbleMotor setIntensity:] in libsdl2_sys-358bf0a65f35dd6c.rlib(SDL_mfijoystick.m.o)
            "_CHHapticEventParameterIDHapticIntensity", referenced from:
                -[SDL_RumbleMotor setIntensity:] in libsdl2_sys-358bf0a65f35dd6c.rlib(SDL_mfijoystick.m.o)
            "_CHHapticEventTypeHapticContinuous", referenced from:
                -[SDL_RumbleMotor setIntensity:] in libsdl2_sys-358bf0a65f35dd6c.rlib(SDL_mfijoystick.m.o)
            "_OBJC_CLASS_$_CHHapticDynamicParameter", referenced from:
                objc-class-ref in libsdl2_sys-358bf0a65f35dd6c.rlib(SDL_mfijoystick.m.o)
            "_OBJC_CLASS_$_CHHapticEvent", referenced from:
                objc-class-ref in libsdl2_sys-358bf0a65f35dd6c.rlib(SDL_mfijoystick.m.o)
            "_OBJC_CLASS_$_CHHapticEventParameter", referenced from:
                objc-class-ref in libsdl2_sys-358bf0a65f35dd6c.rlib(SDL_mfijoystick.m.o)
            "_OBJC_CLASS_$_CHHapticPattern", referenced from:
                objc-class-ref in libsdl2_sys-358bf0a65f35dd6c.rlib(SDL_mfijoystick.m.o)
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)