cmd/cgo: compilation failure on Apple silicon when GOHOSTARCH=arm64 and GOARCH=amd64

[nholstein]

What version of Go are you using (go version)?

$ gotip version
go version devel +3dd5867605 Fri Jan 1 20:05:20 2021 +0000 darwin/arm64

Does this issue reproduce with the latest release?

Yes. (go1.16beta1-42-g3dd5867605)

What operating system and processor architecture are you using (go env)?

CGO_ENABLED=1 GOARCH=amd64 gotip env Output

$ CGO_ENABLED=1 GOARCH=amd64 gotip env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/ngh/Library/Caches/go-build"
GOENV="/Users/ngh/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/ngh/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/ngh/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/ngh/sdk/gotip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/ngh/sdk/gotip/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="devel +3dd5867605 Fri Jan 1 20:05:20 2021 +0000"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/ngh/sdk/gotip/src/cmd/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/7q/4jtm00sd11d6vh2r7m59snmr0000gn/T/go-build3823404109=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I'm running the latest gotip master (go1.16beta1-42-g3dd5867605) built with GOARCH=arm64 on an Apple Silicon laptop. When building an x86_64 executable with CGO_ENABLED=1 GOARCH=amd64 the compilation fails in src/runtime/cgo/gcc_darwin_amd64.c due to Clang's default arm64 target.

This failure is limited to when cgo is enabled; I've seen no issues with the default behavior of CGO_ENABLED=0.

This can be tested by building any executable under the cmd directory:

[ngh@acorn gotip]$ cd src/cmd && CGO_ENABLED=1 GOARCH=amd64 gotip build -o go.amd64 -x ./go
WORK=/var/folders/7q/4jtm00sd11d6vh2r7m59snmr0000gn/T/go-build1701927817
mkdir -p $WORK/b115/
cd /Users/ngh/sdk/gotip/src/runtime/cgo
CGO_LDFLAGS='"-g" "-O2" "-lpthread"' /Users/ngh/sdk/gotip/pkg/tool/darwin_arm64/cgo -objdir $WORK/b115/ -importpath runtime/cgo -import_runtime_cgo=false -import_syscall=false -- -I $WORK/b115/ -g -O2 -Wno-nullability-completeness -Wall -Werror ./cgo.go
cd $WORK
clang -fno-caret-diagnostics -c -x c - -o /dev/null || true
clang -Qunused-arguments -c -x c - -o /dev/null || true
clang -fdebug-prefix-map=a=b -c -x c - -o /dev/null || true
clang -gno-record-gcc-switches -c -x c - -o /dev/null || true
cd $WORK/b115
TERM='dumb' clang -I /Users/ngh/sdk/gotip/src/runtime/cgo -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -fno-common -I ./ -g -O2 -Wno-nullability-completeness -Wall -Werror -o ./_x001.o -c _cgo_export.c
TERM='dumb' clang -I /Users/ngh/sdk/gotip/src/runtime/cgo -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -fno-common -I ./ -g -O2 -Wno-nullability-completeness -Wall -Werror -o ./_x002.o -c cgo.cgo2.c
cd /Users/ngh/sdk/gotip/src/runtime/cgo
TERM='dumb' clang -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -fno-common -I $WORK/b115/ -g -O2 -Wno-nullability-completeness -Wall -Werror -o $WORK/b115/_x003.o -c gcc_context.c
TERM='dumb' clang -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -fno-common -I $WORK/b115/ -g -O2 -Wno-nullability-completeness -Wall -Werror -o $WORK/b115/_x004.o -c gcc_darwin_amd64.c
# runtime/cgo
gcc_darwin_amd64.c:62:15: error: unknown token in expression
<inline asm>:1:12: note: instantiated into assembly here
gcc_darwin_amd64.c:62:15: error: invalid operand
<inline asm>:1:12: note: instantiated into assembly here

What did you expect to see?

Compilation success, with an amd64 executable.

What did you see instead?

A compilation error.

While this may be an uncommon cross-build case, this is necessary to build an executable to run on an Intel Mac from a non-Rosetta2 Apple Mac. I found two potential solutions: manually setting CGO_CFLAGS and CGO_LDFLAGS when running gotip build, and a small patch to cmd/go/internal/work which adds the necessary flags by default.

$ CGO_ENABLED=1 GOARCH=amd64 CGO_CFLAGS="-arch x86_64" CGO_LDFLAGS="-arch x86_64" gotip build -o go.amd64 ./go
$ file ./go.amd64
./go.amd64: Mach-O 64-bit executable x86_64

(The linked patch addresses would also impact other installations when using CC=clang. I haven't tested any combination other than GOOS=darwin GOHOSTARCH=arm64!)

It's worth noting that while I encountered this issue on an Apple Silicon laptop using the default CC=clang, I've used CGO to cross-build on other platforms before, and did not encounter any similar issue. In particular, this seems to be partially caused by support for cross-compilation support in GCC vs. Clang; whereas a single toolchain GCC is built for a specific architecture (perhaps with 32- vs 64-bit support), a single Clang toolchain can support multiple targets. In these other cases I didn't bat an eye when needing to manually configure CC, CGO_CFLAGS, and CGO_LDFLAGS.

While this case is technically cross-building, it felt unexpected that I needed to tweak the behavior here. So this many be nothing more than a bug in my expectations.

[nholstein]

<aside> Thanks to everyone who's put effort into supporting the new Apple processors. I've been amazed at how well things have worked out of the gate for me, and I very much appreciate all your efforts!

[toothrot]

/cc @ianlancetaylor

[bcmills]

CC @cherrymui @jayconrod @matloob

[bcmills]

I'm curious: what happens if you set CC='clang -arch=x86_64'?

[nholstein]

@bcmills, your suggestion works, and is simpler that setting both of CGO_CFLAGS and CGO_LDFLAGS:

[ngh@acorn cmd]$ CC="clang -arch x86_64" CGO_ENABLED=1 GOARCH=amd64 gotip build -o go.amd64 ./go
[ngh@acorn cmd]$ file go.amd64
go.amd64: Mach-O 64-bit executable x86_64

(Note that it's -arch x86_64, not -arch=x86_64; the latter fails.)

[cherrymui]

In general, for cross-compilation (even on the same machine), the C compiler needs to be a cross compiler that targets the target platform, not the host platform. Setting CC (or CGO_CFLAGS, etc.) is the right thing to do, as @bcmills mentioned. (Arguably, the default target selection rule for the system's C compiler on Apple Silicon makes it more confusing and annoying.) As it works, is there anything we still need to do here, or we can close this?

[gopherbot]

Change https://golang.org/cl/283812 mentions this issue: cmd/dist, cmd/go: pass -arch for C compilation on Darwin by default