Compilation fails in GitLab CI

[nixxquality]

I wanted to use this image to automatically build macOS binaries but it's not working out for some reason.

When I ran the image using podman on the same server it does work, though.

$ podman run -it joseluisq/rust-linux-darwin-builder
$ git clone https://gitgud.io/nixx/inquirs/ 
$ cd inquirs
$ cargo build --release --target=x86_64-apple-darwin
# works!

in CI:

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-m64" "-arch" "x86_64" "-L" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.0.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.1.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.10.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.11.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.12.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.13.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.14.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.15.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.2.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.3.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.4.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.5.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.6.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.7.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.8.rcgu.o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.inquirs.46q463ij-cgu.9.rcgu.o" "-o" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/inquirs.sbos325t37vyccn.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps" "-L" "/home/user/nixx/inquirs/target/release/deps" "-L" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/libcrossterm-013f4c15523506cc.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/libsignal_hook-90e3c3f4ba572f80.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/libsignal_hook_registry-93104b4940893217.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/libmio-8598f6ef521bd038.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/liblog-e50ed45a6ece6c8e.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/liblazy_static-a598c89f717ca744.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/libbitflags-5714a0ef175ceec8.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/libparking_lot-ff41adc08c71d02a.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/libparking_lot_core-9c85518863cb4dfe.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/liblibc-71390682de1c85c2.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/libsmallvec-4c513aa56c8b7b2c.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/liblock_api-181162683a0d1810.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/libscopeguard-e6fc67cd518d4401.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/libinstant-f2e0f75766132e58.rlib" "/home/user/nixx/inquirs/target/x86_64-apple-darwin/release/deps/libcfg_if-3786c029dcae5619.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libstd-9225b8e469c32698.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-2fe2b8cd4fa63c34.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libobject-80083cf4ee7ee3ff.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-ddf3832e718029e6.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libgimli-ba92c986e8ef2ca8.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-acd0bd456d8da820.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-55608c4156e4ecbb.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-4be618026614fdc7.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libunwind-fee25922260aa100.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-38ba3c5cc408e14d.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/liblibc-899852a009fbe35f.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/liballoc-5687564a0ccea916.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-9ed2c58b16cff4ee.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libcore-244bf5611f93ff45.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-70df686e6b4ebf5b.rlib" "-liconv" "-lSystem" "-lresolv" "-lc" "-lm" "-liconv"
  = note: cc: error: x86_64: No such file or directory
          cc: error: unrecognized command line option '-arch'; did you mean '-march='?
          
error: aborting due to previous error

https://gitgud.io/nixx/inquirs/-/jobs/155135

# ci configuration

x86_64-apple-darwin:
  stage: build
  image: joseluisq/rust-linux-darwin-builder
  artifacts:
    paths:
      - target/x86_64-apple-darwin/release/$CI_PROJECT_NAME
  script:
    - cargo build --release --target=x86_64-apple-darwin

I hope you can at least give me some advice on why this may be happening.

[joseluisq]

Kind of weird. For example this Docker image project makes also CI via Github Actions + Docker just performing a simple cross compilation test with no issues.

Did you indicate the CC and CXX environment variables too? Maybe some of your crates requires so?
The idea is to prefer o64-clang and o64-clang++ part of osxcross/target/bin because the -arch option is part of the Apple extensions to gcc.
More here.

[nixxquality]

There's no native code so CC/CXX shouldn't have been required. What's confusing me is that it works in docker but the exact same(?) command fails in CI.

Here's a job I did with CC/CXX set, printing out env before the run, and noting that it's using the correct o64-clang executables.

Maybe you can spot something odd in here?

https://gitgud.io/nixx/inquirs/-/jobs/155155

[joseluisq]

It looks like your pipeline setup is fine and yeah your project doesn't require C linkage.
However I don't really get why is resulting in a linking error.

So I did a quick test on my Drone CI to try to reproduce it but I couldn't.
My Drone CI and pipeline have nothing more than this simple setup.

---
kind: pipeline
type: docker
name: development
platform:
  os: linux
  arch: amd64
steps:
- name: restore-cache
  image: drillster/drone-volume-cache
  volumes:
    - name: cache
      path: /cache
  settings:
    restore: true
    mount:
      - ./target

- name: test
  image: joseluisq/rust-linux-darwin-builder:1.52.1
  commands:
    - env
    - which o64-clang
    - which o64-clang++
    - cargo build --release --target=x86_64-apple-darwin

- name: rebuild-cache
  image: drillster/drone-volume-cache
  volumes:
    - name: cache
      path: /cache
  settings:
    rebuild: true
    mount:
      - ./target
volumes:
  - name: cache
    host:
      path: /tmp/cache

No explicit CC or CXX envs needed. You can also just ignore the cache steps.
If you see the test step is just identical to yours.

Perhaps the issue relates to your CI cache mechanism, lack of privileges (not sure really) or maybe you are facing something like this issue. But unfortunately I'm just guessing.

Anyway since I can not reproduce the issue via Docker on Drone CI or Github Actions. And since it seems not directly related to this Docker image, I'm going to close the issue for now. However feel free to re-open it if needed.

[yq314]

I face the exact same problem, it works in docker but fails in Gitlab CI with the same error. I am specifying CC=o64-clang CXX=o64-clang++

[joseluisq]

I will try to give it a second look at this. I will let you know here.

[yq314]

Steps to reproduce:

brew install gitlab-runner
brew services start gitlab-runner

git clone https://gitgud.io/nixx/inquirs/
cd inquires
gitlab-runner exec docker x86_64-apple-darwin

[nixxquality]

For what it's worth, I did open an issue on the GitLab tracker but it has seen no activity since.
https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27966

[joseluisq]

I addressed the "issue" that was really not but instead an extra line missing to append to your pipeline scripts before cross-compile your program.

~> gitlab-runner exec docker x86_64-apple-darwin
Runtime platform                                    arch=amd64 os=linux pid=28513 revision=8925d9a0 version=14.1.0
Running with gitlab-runner 14.1.0 (8925d9a0)
Preparing the "docker" executor
Using Docker executor with image joseluisq/rust-linux-darwin-builder:1.54.0 ...
Authenticating with credentials from /home/joseluisq/.docker/config.json
Pulling docker image joseluisq/rust-linux-darwin-builder:1.54.0 ...
Using docker image sha256:e808e07975ccc5746a678ad672b6b5330cbdc73b919483406b4378710513ebd5 for joseluisq/rust-linux-darwin-builder:1.54.0 with digest joseluisq/rust-linux-darwin-builder@sha256:67afeff25184a518dd047127f320ec417ce3b773706cddfd2efd10f04ca7aa7f ...
Preparing environment
Running on runner--project-0-concurrent-0 via quintana...
Getting source from Git repository
Fetching changes...
Initialized empty Git repository in /builds/project-0/.git/
Created fresh repository.
Checking out d0a8fdfe as master...

Skipping Git submodules setup
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
Restoring cache
Checking cache for one-key-to-rule-them-all...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
Using docker image sha256:e808e07975ccc5746a678ad672b6b5330cbdc73b919483406b4378710513ebd5 for joseluisq/rust-linux-darwin-builder:1.54.0 with digest joseluisq/rust-linux-darwin-builder@sha256:67afeff25184a518dd047127f320ec417ce3b773706cddfd2efd10f04ca7aa7f ...
$ rustc --version && cargo --version
rustc 1.54.0 (a178d0322 2021-07-26)
cargo 1.54.0 (5ae8d74b3 2021-06-22)
$ cp /root/.cargo/config "$CARGO_HOME/"
$ cargo build --release --target=x86_64-apple-darwin
    Updating crates.io index
 Downloading crates ...
  Downloaded signal-hook v0.1.17
  Downloaded crossterm v0.18.2
  Downloaded lazy_static v1.4.0
  Downloaded log v0.4.14
  Downloaded parking_lot_core v0.8.3
  Downloaded mio v0.7.13
  Downloaded lock_api v0.4.4
  Downloaded bitflags v1.3.2
  Downloaded instant v0.1.10
  Downloaded scopeguard v1.1.0
  Downloaded parking_lot v0.11.1
  Downloaded cfg-if v1.0.0
  Downloaded smallvec v1.6.1
  Downloaded signal-hook-registry v1.4.0
  Downloaded libc v0.2.99
   Compiling libc v0.2.99
   Compiling cfg-if v1.0.0
   Compiling log v0.4.14
   Compiling smallvec v1.6.1
   Compiling scopeguard v1.1.0
   Compiling bitflags v1.3.2
   Compiling lazy_static v1.4.0
   Compiling instant v0.1.10
   Compiling lock_api v0.4.4
   Compiling signal-hook-registry v1.4.0
   Compiling parking_lot_core v0.8.3
   Compiling mio v0.7.13
   Compiling parking_lot v0.11.1
   Compiling signal-hook v0.1.17
   Compiling crossterm v0.18.2
   Compiling inquirs v0.1.0 (/builds/project-0)
    Finished release [optimized] target(s) in 38.89s
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
Saving cache for successful job
Creating cache one-key-to-rule-them-all...
WARNING: /builds/project-0/.cargo/bin: no matching files 
/builds/project-0/.cargo/registry/index: found 84 matching files and directories 
/builds/project-0/.cargo/registry/cache: found 17 matching files and directories 
WARNING: /builds/project-0/.cargo/git/db: no matching files 
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally. 
Created cache
Job succeeded

and the Gitlab pipeline used:

x86_64-apple-darwin:
  stage: build
  image: joseluisq/rust-linux-darwin-builder:1.54.0
  artifacts:
    paths:
      - target/x86_64-apple-darwin/release/$CI_PROJECT_NAME
  script:
    - cp /root/.cargo/config "$CARGO_HOME/"
    - cargo build --release --target=x86_64-apple-darwin

The "problem" was that your Gitlab pipeline uses a custom working directory that obviously is lacking some setup necessary to tell Cargo about the right linker to use for the particular target (x86_64-apple-darwin). That's why the linkage error.
So copying the container default Cargo config of this Docker image to your new CARGO_HOME made the trick.

Of course this is not an issue with this Docker image however clarifying it on the README file is not bad idea.
So PR welcome.

[nixxquality]

Thank you for the thorough investigation.

I guess the culprit was that I redefined CARGO_HOME in order to retain cache between builds.
Removing that also fixes the build.

I'm not too familiar with the Cargo toolchain, I assume there's no way to have multiple CARGO_HOMEs?
Or at least look for a config file in multiple locations?

[joseluisq]

Looks like to have multiple Cargo homes is not possible.
But I think is up-to the user (dev) to decide how CARGO_HOME should behave or not.