cmd/link: cgo cross linking using the clang --target flag creates crashing programs

[apsaltis-ddog]

Go version

go version go1.24.2 linux/amd64

Output of go env in your module/workspace:

(Some output here and below manually modified to hide internal configuration)

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='$HOME/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='$HOME/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build353822596=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='$HOME/tmp/go-cross/go.mod'
GOMODCACHE='$HOME/go/pkg/mod'
GONOPROXY=''
GONOSUMDB='$INTERNAL'
GOOS='linux'
GOPATH='$HOME/go'
GOPRIVATE=''
GOPROXY='$INTERNAL,direct'
GOROOT='$HOME/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='$HOME/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='$HOME/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.2'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Attempted to cross-compile a cgo binary from amd64 to arm64 using a private LLVM/Clang-based toolchain and sysroot. I was able to reproduce this from arm64 crossing to amd64 as well.

Minimal example (which doesn't run in the playground due to needing cgo): https://go.dev/play/p/a0qGj4Vakf-

Built with:

CGO_ENABLED=1 GOOS=linux GOARCH=arm64 CC=/path/to/private/clang CGO_CFLAGS="--sysroot=/path/to/private/sysroot-arm --target=aarch64-unknown-linux-gnu" \
    go build \
        -ldflags="-v -extld C/path/to/private/clang -extldflags '-fuse-ld=lld --sysroot=/path/to/private/sysroot-arm --target=aarch64-unknown-linux-gnu --verbose'" \
        -gcflags=-v

The output binary can be run on an ARM host, or via qemu with something like:

qemu-aarch64 -L /path/to/private/sysroot-arm ./go-cross

What did you see happen?

The go tool link output from the above go build command contained a linker invocation like this:

(The "host link" outputs below are particularly long, and is more readable in an editor)

host link: "/path/to/private/clang" "-o" "/tmp/go-build481106842/b001/exe/a.out" "-rdynamic" "/tmp/go-link-168810886/go.o" "/tmp/go-link-168810886/000000.o" "/tmp/go-link-168810886/000001.o" "/tmp/go-link-168810886/000002.o" "/tmp/go-link-168810886/000003.o" "/tmp/go-link-168810886/000004.o" "/tmp/go-link-168810886/000005.o" "/tmp/go-link-168810886/000006.o" "/tmp/go-link-168810886/000007.o" "/tmp/go-link-168810886/000008.o" "/tmp/go-link-168810886/000009.o" "/tmp/go-link-168810886/000010.o" "/tmp/go-link-168810886/000011.o" "/tmp/go-link-168810886/000012.o" "/tmp/go-link-168810886/000013.o" "/tmp/go-link-168810886/000014.o" "/tmp/go-link-168810886/000015.o" "-O2" "-g" "-O2" "-g" "-lpthread" "-fuse-ld=lld" "--sysroot=/path/to/private/sysroot-arm" "--target=aarch64-unknown-linux-gnu" "--verbose"

If I use the same compiler targeting the host architecture, using a sysroot compiled for it (in this case amd64/x86_64), the output is different:

CGO_ENABLED=1 GOOS=linux GOARCH=amd64 CC=/path/to/private/clang CGO_CFLAGS="--sysroot=/path/to/private/sysroot-amd64 --target=x86_64-unknown-linux-gnu" \
    go build \
        -ldflags="-v -extld C/path/to/private/clang -extldflags '-fuse-ld=lld --sysroot=/path/to/private/sysroot-amd64 --target=x86_64-unknown-linux-gnu --verbose'" \
        -gcflags=-v

...

host link: "$HOME/tmp/llvm-17.0.6-dist-x86_64/bin/clang" "-m64" "-Wl,--build-id=0xe3c881e6feaf37cd5c2a7e4e643c549ff859ee17" "-o" "/tmp/go-build3873136084/b001/exe/a.out" "-Wl,--export-dynamic-symbol=_cgo_panic" "-Wl,--export-dynamic-symbol=_cgo_topofstack" "-Wl,--export-dynamic-symbol=crosscall2" "-Qunused-arguments" "-Wl,--compress-debug-sections=zlib" "/tmp/go-link-1498717480/go.o" "/tmp/go-link-1498717480/000000.o" "/tmp/go-link-1498717480/000001.o" "/tmp/go-link-1498717480/000002.o" "/tmp/go-link-1498717480/000003.o" "/tmp/go-link-1498717480/000004.o" "/tmp/go-link-1498717480/000005.o" "/tmp/go-link-1498717480/000006.o" "/tmp/go-link-1498717480/000007.o" "/tmp/go-link-1498717480/000008.o" "/tmp/go-link-1498717480/000009.o" "/tmp/go-link-1498717480/000010.o" "/tmp/go-link-1498717480/000011.o" "/tmp/go-link-1498717480/000012.o" "/tmp/go-link-1498717480/000013.o" "/tmp/go-link-1498717480/000014.o" "/tmp/go-link-1498717480/000015.o" "-O2" "-g" "-O2" "-g" "-lpthread" "-no-pie" "-fuse-ld=lld" "--sysroot=/path/to/private/sysroot-amd64" "--target=x86_64-unknown-linux-gnu" "--verbose"

Both commands succeed, and produce a binary.

These flag differences are numerous and consequential: running the cross-compiled binary (in this particular case, using qemu) crashes at startup like this:

> qemu-aarch64 -L /path/to/private/sysroot-arm ./go-cross
runtime: pcHeader: magic= 0xfffffff1 pad1= 0 pad2= 0 minLC= 4 ptrSize= 8 pcHeader.textStart= 0xd9d20 text= 0x7f051fca9d20 pluginpath=
fatal error: invalid function symbol table
runtime: panic before malloc heap initialized

runtime stack:
runtime.throw({0x7f051fbff423?, 0x0?})
	$HOME/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/panic.go:1101 +0x38 fp=0x4000008000b0 sp=0x400000800080 pc=0x7f051fd11208
runtime.moduledataverify1(0x7f051fd8d5e0?)
	$HOME/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/symtab.go:623 +0x644 fp=0x4000008001d0 sp=0x4000008000b0 pc=0x7f051fd13704
runtime.moduledataverify(...)
	$HOME/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/symtab.go:599
runtime.schedinit()
	$HOME/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/proc.go:834 +0x88 fp=0x400000800260 sp=0x4000008001d0 pc=0x7f051fce2fd8
runtime.rt0_go()
	$HOME/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/asm_arm64.s:86 +0xa4 fp=0x400000800290 sp=0x400000800260 pc=0x7f051fd15154

The binary fails similarly on an actual ARM host as well. When I investigated this, it seems that the critical difference is the -no-pie flag in the second (amd64, same as the build host) output, as adding -buildmode pie to the cross-compiling go build prevents the binary from crashing like this.

What did you expect to see?

The flags in the two "host link" invocations above should be more similar (aside from arch-specific flags), and the cross-compiled program should not crash as mentioned above when run in an emulator or a system with that architecture.

[apsaltis-ddog]

I think the underlying problem is in how go tool link is doing tests to determine what flags it can pass to the underlying linker. While exploring, I found trimLinkerArgv in the linker code. Notably, it contains, -target in a few places, but doesn't mention --target anywhere.

Clang supports both ways of specifying targets, but they are used differently: -target only expects the target in a second arg, and --target only expects it in the same arg (--target=$ARG).

From the looks of it, in this list:

go/src/cmd/link/internal/ld/lib.go

Lines 2186 to 2195 in 2cb9e7f

	prefixesToKeep := []string{
	"-f",
	"-m",
	"-p",
	"-Wl,",
	"-arch",
	"-isysroot",
	"--sysroot",
	"-target",
	}

-target should be --target, or --target should be added to the list. I'd be happy to create a PR that would do this -- this created non-crashing binaries for me.

[gabyhelp]

Related Issues

• runtime/cgo: can't cross compile from os x to linux with cgo #12888 (closed)
• Crosscompile with race detector fails with fault and panic #49102 (closed)
• Error building a simple program with GOOS=darwin GOARCH=arm64 environment variables #39033 (closed)
• cmd/link: function "decodetypeGcmask" read gcdata from shared object file may not correct when use `lld` link in linux/arm64 #69466
• net: program cross-compiled for darwin/arm64 from linux/amd64 is SIGKILLed since CL 446178 #56599 (closed)
• cmd/go: cross-compiling for Linux x86_64 with CGO -race enabled failed on macOS Mojave (Darwin) #30863 (closed)
• cross compiling issue #66642 (closed)
• cmd/buildid: failures in cmd/go introduced in commit afd090c on multiple platforms including ppc64le, s390, and arm64 #23339 (closed)
• cmd/link: cannot cross compile a linux plugin on macOS #22462

Related Discussions

• Facing linker issues while cross-compiling from Ubuntu/amd64 to darwin/amd64

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[mknyszek]

@apsaltis-ddog Your fix seems reasonable. Thanks for investigating. Feel free to send a patch!

[gopherbot]

Change https://go.dev/cl/697340 mentions this issue: cmd/link: ensure that the "--target" clang flag is considered when testing for C linker capabilities

[apsaltis-ddog]

Looks like https://go.dev/cl/703055 addressed this before https://go.dev/cl/697340 could be merged in. Should this be closed?