Cross-compiling from x86_64 to arm64 on macOS 11.1 fails to compile stage1 std artifacts

[felixbuenemann]

When cross-compiling either v1.48 or master@d806d656578 from x86_64 to arm64 on MacBook Air (M1, 2020) running macOS 11.1 Beta (20C5048l) and Xcode 12.3 beta (12C5020f), the compilation fails when trying to build the stage1 std artifacts for stage2:

% uname -m
arm64
% grep -E "^(build|host|target) =" config.toml
build = 'x86_64-apple-darwin'
host = ['aarch64-apple-darwin']
target = ['aarch64-apple-darwin']

% make
    Finished dev [unoptimized + debuginfo] target(s) in 0.14s
warning: there have been changes to x.py since you last updated.
help: consider looking at the changes in `src/bootstrap/CHANGELOG.md`
note: to silence this warning, update `config.toml` to use `changelog-seen = 2` instead
Building stage0 std artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.11s
Copying stage0 std from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Could not determine the LLVM submodule commit hash. Assuming that an LLVM rebuild is not necessary.
To force LLVM to rebuild, remove the file `/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/llvm/llvm-finished-building`
Building stage0 compiler artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.18s
Copying stage0 rustc from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Assembling stage1 compiler (x86_64-apple-darwin)
Building stage1 std artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.13s
Copying stage1 std from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage1 compiler artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.18s
Copying stage1 rustc from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Assembling stage2 compiler (x86_64-apple-darwin)
Building stage1 std artifacts (x86_64-apple-darwin -> aarch64-apple-darwin)
   Compiling std v0.0.0 (/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/library/std)
error: linking with `clang` failed: exit code: 1
  |
  = note: "clang" "-arch" "arm64" "-L" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.0.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.1.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.10.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.11.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.12.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.13.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.14.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.15.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.2.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.3.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.4.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.5.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.6.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.7.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.8.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.std.df713kw9-cgu.9.rcgu.o" "-o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/libstd-e2d6f492c8b2d1df.dylib" "-Wl,-exported_symbols_list,/private/tmp/rustcCncuXN/list" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.s8krdvcbzje2gsl.rcgu.o" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps/std-e2d6f492c8b2d1df.32vp5h2h70z0baum.rcgu.o" "-Wl,-dead_strip" "-dynamiclib" "-Wl,-dylib" "-Wl,-install_name" "-Wl,@rpath/libstd-e2d6f492c8b2d1df.dylib" "-nodefaultlibs" "-L" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/deps" "-L" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/release/deps" "-L" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1-std/aarch64-apple-darwin/release/build/compiler_builtins-c337dd450e333db0/out" "-L" "/private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/build/x86_64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib" "-lSystem" "-lresolv" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/libpanic_unwind-ae9f774b49d93062.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/libobject-8856f1ded0536c74.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/libaddr2line-0f0383ea697c7750.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/libgimli-92cf120f7800af66.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/librustc_demangle-a7a71dbb2e687bfa.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/libhashbrown-8540130587673a3f.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/librustc_std_workspace_alloc-6b033095361c6f7b.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/libunwind-38f331437c8ebca7.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/libcfg_if-ffd487bab818ca85.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/liblibc-e8ca45c499b5b583.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/liballoc-7869d17a01843983.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/librustc_std_workspace_core-e5304dd642386a97.rlib" "-Wl,-force_load" "-Wl,/private/tmp/rustcCncuXN/libcore-c584be2e4a499900.rlib" "/private/tmp/rustcCncuXN/libcompiler_builtins-235bdf94dbd2400b.rlib" "-lc" "-lm" "-Wl,-rpath,@loader_path/../lib"
  = note: Undefined symbols for architecture x86_64:
            "std::net::udp::UdpSocket::set_multicast_loop_v4::h29c5f9a7502375e9", referenced from:
               -exported_symbol[s_list] command line option
            "std::fs::Permissions::set_readonly::hca79eaf67028178b", referenced from:
               -exported_symbol[s_list] command line option
            "std::io::stdio::Stdin::lock::h7f81bb5795d87a3d", referenced from:
               -exported_symbol[s_list] command line option
            "std::net::ip::Ipv4Addr::is_shared::h37f841feb8719198", referenced from:
               -exported_symbol[s_list] command line option
...

Thousands of similar log lines follow with all the different symbols.

Not sure why it is expecting x86_64 symbols, since the objects are arm64, as verified with lips -archs some.o.

Build Tools:

% ninja --version
1.10.1
% clang -v
Apple clang version 12.0.0 (clang-1200.0.32.28)
Target: arm64-apple-darwin20.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
% ld -v
@(#)PROGRAM:ld  PROJECT:ld64-609.8
BUILD 19:59:02 Nov  4 2020
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
LTO support using: LLVM version 12.0.0, (clang-1200.0.32.28) (static support for 27, runtime is 27)
TAPI support using: Apple TAPI version 12.0.0 (tapi-1200.0.23.5)

Let me know if you need additional info.

[camelid]

@felixbuenemann Hmm, why are you running make and not x.py directly? Can you share the Makefile you are running?

[felixbuenemann]

I'm using the default Makefile that got generated from running:

% ./configure --prefix=/opt/homebrew/Cellar/rust/1.48.0 --release-channel=stable --build=x86_64-apple-darwin --host=aarch64-apple-darwin --target=aarch64-apple-darwin
configure: processing command line
configure:
configure: install.prefix       := /opt/homebrew/Cellar/rust/1.48.0
configure: rust.channel         := stable
configure: build.build          := x86_64-apple-darwin
configure: build.host           := ['aarch64-apple-darwin']
configure: build.target         := ['aarch64-apple-darwin']
configure: build.configure-args := ['--prefix=/opt/homebrew/Cellar/rust/1.48.0',  ...
configure:
configure: writing `config.toml` in current directory
configure:
configure: run `python /private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/x.py --help`
configure:

I am inside a build of the rust Hombre formula that has been modified to add the cross-compile configure arguments, but the same happened in a manual build with a checkout on master.

Here are the contents:

ifdef VERBOSE
Q :=
BOOTSTRAP_ARGS := -v
else
Q := @
BOOTSTRAP_ARGS :=
endif

BOOTSTRAP := /opt/homebrew/opt/python@3.9/bin/python3.9 /private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/src/bootstrap/bootstrap.py

all:
	$(Q)$(BOOTSTRAP) build --stage 2 $(BOOTSTRAP_ARGS)
	$(Q)$(BOOTSTRAP) doc --stage 2 $(BOOTSTRAP_ARGS)

help:
	$(Q)echo 'Welcome to the rustbuild build system!'
	$(Q)echo
	$(Q)echo This makefile is a thin veneer over the ./x.py script located
	$(Q)echo in this directory. To get the full power of the build system
	$(Q)echo you can run x.py directly.
	$(Q)echo
	$(Q)echo To learn more run \`./x.py --help\`

clean:
	$(Q)$(BOOTSTRAP) clean $(BOOTSTRAP_ARGS)

rustc-stage1:
	$(Q)$(BOOTSTRAP) build --stage 1 library/test $(BOOTSTRAP_ARGS)
rustc-stage2:
	$(Q)$(BOOTSTRAP) build --stage 2 library/test $(BOOTSTRAP_ARGS)

docs: doc
doc:
	$(Q)$(BOOTSTRAP) doc --stage 2 $(BOOTSTRAP_ARGS)
nomicon:
	$(Q)$(BOOTSTRAP) doc --stage 2 src/doc/nomicon $(BOOTSTRAP_ARGS)
book:
	$(Q)$(BOOTSTRAP) doc --stage 2 src/doc/book $(BOOTSTRAP_ARGS)
standalone-docs:
	$(Q)$(BOOTSTRAP) doc --stage 2 src/doc $(BOOTSTRAP_ARGS)
check:
	$(Q)$(BOOTSTRAP) test --stage 2 $(BOOTSTRAP_ARGS)
check-aux:
	$(Q)$(BOOTSTRAP) test --stage 2 \
		src/tools/cargo \
		src/tools/cargotest \
		$(BOOTSTRAP_ARGS)
check-bootstrap:
	$(Q)/opt/homebrew/opt/python@3.9/bin/python3.9 /private/tmp/rust-20201121-94992-iuj811/rustc-1.48.0-src/src/bootstrap/bootstrap_test.py
dist:
	$(Q)$(BOOTSTRAP) dist $(BOOTSTRAP_ARGS)
distcheck:
	$(Q)$(BOOTSTRAP) dist $(BOOTSTRAP_ARGS)
	$(Q)$(BOOTSTRAP) test --stage 2 distcheck $(BOOTSTRAP_ARGS)
install:
	$(Q)$(BOOTSTRAP) install $(BOOTSTRAP_ARGS)
tidy:
	$(Q)$(BOOTSTRAP) test --stage 2 src/tools/tidy $(BOOTSTRAP_ARGS)
prepare:
	$(Q)$(BOOTSTRAP) build --stage 2 nonexistent/path/to/trigger/cargo/metadata

check-stage2-T-arm-linux-androideabi-H-x86_64-unknown-linux-gnu:
	$(Q)$(BOOTSTRAP) test --stage 2 --target arm-linux-androideabi
check-stage2-T-x86_64-unknown-linux-musl-H-x86_64-unknown-linux-gnu:
	$(Q)$(BOOTSTRAP) test --stage 2 --target x86_64-unknown-linux-musl

TESTS_IN_2 := \
	src/test/ui \
	src/test/compile-fail \
	src/tools/linkchecker

ci-subset-1:
	$(Q)$(BOOTSTRAP) test --stage 2 $(TESTS_IN_2:%=--exclude %)
ci-subset-2:
	$(Q)$(BOOTSTRAP) test --stage 2 $(TESTS_IN_2)

TESTS_IN_MINGW_2 := \
	src/test/ui \
	src/test/compile-fail

ci-mingw-subset-1:
	$(Q)$(BOOTSTRAP) test --stage 2 $(TESTS_IN_MINGW_2:%=--exclude %)
ci-mingw-subset-2:
	$(Q)$(BOOTSTRAP) test --stage 2 $(TESTS_IN_MINGW_2)


.PHONY: dist

[felixbuenemann]

@camelid I was able to get past the error by running arch -x86_64 make, so all the build tools run in Rosetta 2 emulation.

[felixbuenemann]

I'm not sure if this is a user error or a build system error.

[felixbuenemann]

Related to #73908.

[shepmaster]

In general, we actively exercise cross-compiling from x86_64 to aarch64 because that's what the current CI does. Perhaps you can compare what you do vs what we do.

I wouldn't expect 1.48 to work, but beta and nightly should be fine.

[shepmaster]

However, there are issues with macOS about running the compiler tools in Rosetta2 and code signing:

• Why does my native arm64 application built using an x86_64 build system fail to be code signed unless I remove the previous executable?

One solution I've used is to override cc etc. with a small script that forces it back to arm64:

% cat ~/bin/force-arm64-cc
#!/usr/bin/env bash

exec arch -arm64e cc "$@"