cmd/compile: -buildmode=c-archive produces code not suitable for use in a shared object on arm64

[nmeum]

tl;dr On aarch64, Go 1.21 seems to emit code that is not suitable for use in a C/C++ shared library with -buildmode=c-archive on aarch64. I believe this to be a Go 1.21 regression as this was explicitly mentioned as a supported feature in commit d92a360. The root cause seems to be a R_AARCH64_ADR_PREL_PG_HI21 relocation for the crosscall2 symbol.

What version of Go are you using (go version)?

$ go version
go version go1.21.1 linux/arm64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/home/soeren/.cache/go-build'
GOENV='/home/soeren/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS='-buildmode=pie -modcacherw -trimpath -buildvcs=false'
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/soeren/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/soeren/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.21.1'
GCCGO='gccgo'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2174769471=/tmp/go-build -gno-record-gcc-switches

What did you do?

I maintain the Go package for Alpine Linux (which uses musl libc) with the release of Go 1.21.1 we rebuild all of our packaged Go software for the CVE fixes in the standard library. While doing so, we noticed that the package fcitx5-bamboo failed to build from source on aarch64 (on all other architecture it build fine). The error message was:

/usr/lib/gcc/aarch64-alpine-linux-musl/13.1.1/../../../../aarch64-alpine-linux-musl/bin/ld: bamboo/bamboo-core.a(go.o): relocation R_AARCH64_ADR_PREL_PG_HI21 against symbol `crosscall2' which may bind externally can not be used when making a shared object; recompile with -fPIC
bamboo/bamboo-core.a(go.o): in function `runtime/cgo.set_crosscall2.abi0':
go.go:(.text+0x8f228): dangerous relocation: unsupported relocation
collect2: error: ld returned 1 exit status
ninja: subcommand failed

The package previously compiled fine with Go 1.20.6 on aarch64.

Closer investigation revealed that the fcitx5-bamboo build system does the following:

1. It compiles a few Go source files to a C archive (bamboo-core.a) using go -buildmode=c-archive.
2. It compiles some C++ code and links it together with the c-archive, emitted by the Go compiler in the previous step, to a shared library using g++ -shared.

Naturally, the error message above is emitted by g++ not by the Go compiler. However, as per a prior discussion on golang-dev and commit d92a360, I believe that files produced by go build -buildmode=c-archive are supposed to be suitable for use in a shared library which the file emitted here (bamboo-core.a) is not. If my assumption is correct, then I believe this to be a regression introduced in Go 1.21, i.e. a bug in the Go compiler. The issue in this regard seems to be a R_AARCH64_ADR_PREL_PG_HI21 relocation emitted for crosscall2:

$ readelf -W -a bamboo/bamboo-core.a | grep ' crosscall2' | grep R_AARCH64_ADR_PREL_PG_HI21
000000000008f228  00000c0000000113 R_AARCH64_ADR_PREL_PG_HI21 000000000008f240 crosscall2 + 0

I am not familiar enough with aarch64 internals and relocations in general in order to determine why this code requires a R_AARCH64_ADR_PREL_PG_HI21 relocation and why this relocation is not suitable for use in a shared library.

To reproduce this run the following commands on aarch64:

$ git clone https://github.com/fcitx/fcitx5-bamboo
$ cd fcitx5-bamboo
$ git checkout 56cd63fac51a1da334903b06875d71a6ce0971de
$ git submodule update --init
$ cd bamboo/
$ go build -o bamboo-core.a -buildmode=c-archive *.go
$ readelf -W -a bamboo-core.a | grep ' crosscall2' | grep R_AARCH64_ADR_PREL_PG_HI21
000000000008f228  00000c0000000113 R_AARCH64_ADR_PREL_PG_HI21 000000000008f240 crosscall2 + 0

This should print the R_AARCH64_ADR_PREL_PG_HI21 relocation, or at least it does on Alpine Linux Edge aarch64. Again, I am not sure why this relocation is not suitable for use in a shared library. If you want to re-produce the g++ error message follow the upstream build instructions. Unfortunately, I did not manage to come up with a minimal example yet.

Full build log for our failing aarch64 Alpine Linux Edge build: fcitx5-bamboo-go-1.21.1-alpine-linux-edge-aarch64.txt

What did you expect to see?

Successful linking of the c-archive emitted by go build -buildmode=c-archive to a shared library with g++ -shared.

What did you see instead?

The g++ error message regarding a R_AARCH64_ADR_PREL_PG_HI21 relocation.

[thanm]

Possibly related: https://sourceware.org/bugzilla/show_bug.cgi?id=28370 ?

[cherrymui]

I think this may be because we take the address of a function without going through the GOT, and crosscall2 happens to be an exported function (exported in the sense of DSO) (https://cs.opensource.google/go/go/+/master:src/runtime/cgo/callbacks.go;l=26)

We probably could bypass the issue by using a local wrapper function.

(This comment is interesting

// We need to export the symbol crosscall2 in order to support
// callbacks from shared libraries. This applies regardless of
// linking mode.

Why it needs to be exported if we're not using -buildmode=shared or -linkshared? In exe, c-archive, or c-shared build modes I think the crosscall2 call are made from cgo-generated code, which is in the same DSO. Maybe SWIG calls can be in another DSO?)

[cherrymui]

@nmeum what version of the C linker do you use? Could you show the output of ld -v?

On my machine with GNU ld (GNU Binutils for Debian) 2.40.50.20230611 it seems to work okay with that particular relocation.

[ianlancetaylor]

At the time the "We need to export the symbol crosscall2" comment was written, SWIG (and cgo) worked by writing a shared library. Now that that is no longer can true we can retire the comment.

[nmeum]

@nmeum what version of the C linker do you use? Could you show the output of ld -v?

On my machine with GNU ld (GNU Binutils for Debian) 2.40.50.20230611 it seems to work okay with that particular relocation.

I am using GNU ld (GNU Binutils) 2.41. However, even with Binutils 2.41 it works when using Go 1.20.7 and breaks with Go 1.21.1.

[nmeum]

So did some further tests it seems that the regression was introduced in c426c87 (this commit also introduced the set_crosscall2 function mentioned in the linker error) prior to this commit (i.e. 2693ade) fcitx5-bamboo builds fine and there is no relocation error.