cmd/link: compilation failure on armv6/armv7 due to truncated relocations

[tpaschalis]

What version of Go are you using (go version)?

$ docker run grafana/agent-build-image:0.21.0 bash -c "go version"
go version go1.20 linux/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

The following is for the build container that cross-compiles to armv6/armv7 and encounters the failure

go env Output

$ docker run grafana/agent-build-image:0.21.0 bash -c "go env"
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm64"
GOVCS=""
GOVERSION="go1.20"
GCCGO="gccgo"
AR="ar"
CC="viceroycc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1972461801=/tmp/go-build -gno-record-gcc-switches"

What did you do?

1. Cloned the grafana/agent repo, and checked out the reproduce-relocation-truncated branch or grafana/agent@7e4cae2
2. Run USE_CONTAINER=1 GOOS=linux GOARCH=arm GOARM=6 make agentctl to cross-compile the agentctl binary for ARMv6
3. Wait around

What did you expect to see?

A successful compilation

What did you see instead?

# github.com/grafana/agent/cmd/grafana-agentctl
/usr/local/go/pkg/tool/linux_amd64/link: running viceroycc failed: exit status 1
/tmp/go-link-3047421554/go.o: in function `github.com/google/gnostic/openapiv3.NewResponse':
/go/pkg/mod/github.com/google/gnostic@v0.6.9/openapiv3/OpenAPIv3.go:3631:(.text+0x207d790): relocation truncated to fit: R_ARM_CALL against `runtime.duffzero'
/go/pkg/mod/github.com/google/gnostic@v0.6.9/openapiv3/OpenAPIv3.go:3635:(.text+0x207d894): relocation truncated to fit: R_ARM_CALL against `runtime.duffzero'
collect2: error: ld returned 1 exit status

make: *** [Makefile:195: agentctl] Error 1

Background

Hello team! I'm opening this issue on behalf of the Grafana Agent squad.

Recently, we saw our ARMv6 and ARMv6 builds start to fail with relocation truncated to fit: errors.

We think it has to do with the growing binary size and the number of dependencies that the Agent brings in. Looking at the GCC arm-specific options, we used the -mlong-calls flag for our ARM builds which hid the issue for a few more commits, until it resurfaced.

We also discovered #15823 for a similar issue around builds, but for a different architecture ppc64le which was fixed at the language level in CL27790. Do you think this is similar enough?

Some other information that might be useful:

• Our process uses a Docker container to perform the build and copies the resulting build back to the host
• It looks like that having CGO enabled/disabled has an effect on builds; we need CGO_ENABLED=1 for our dependencies
• Within the build container, CC is set to a shell script which looks for an appropriate gcc for the GOOS/GOARCH/GOARM tuple and calls out to it; in this case arm-linux-gnueabi-gcc installed from Debian bullseye is used

[tpaschalis]

So the reference PR is over 6 years old now, and things have moved around in the linker.

I noticed the splitTextSections helper; could it be that we need to add sys.ARM to the architectures that need their text sections split, like this? I'd love feedback from anyone more knowledgeable about ARM and the linker here.

[dr2chase]

@golang/compiler

[liggitt]

I'd be interested to know if the workaround in #58425 (comment) works here as well

[rfratto]

👋 Reporting in for the Grafana Agent team while @tpaschalis is out:

I just tested the workaround from #58425 (comment). One of our binaries still gets linking errors for GOARCH=arm GOARM=7 even when using the workaround:

$ GOOS=linux GOARCH=arm GOARM=7 CGO_ENABLED=1 GOEXPERIMENT=nounified \
  go build \
    -ldflags "-X github.com/grafana/agent/pkg/build.Branch=main -X github.com/grafana/agent/pkg/build.Version=main-b82b26c -X github.com/grafana/agent/pkg/build.Revision=b82b26cb -X github.com/grafana/agent/pkg/build.BuildUser=root@bca20e00e921 -X github.com/grafana/agent/pkg/build.BuildDate=2023-02-11T20:15:03Z" \
    -tags "netgo " \
    -gcflags=all=-d=inlstaticinit=0 \
    -o build/grafana-agentctl \
    ./cmd/grafana-agentctl

...

/usr/local/go/pkg/tool/linux_amd64/link: running viceroycc failed: exit status 1
/tmp/go-link-201436594/go.o: in function `k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json.(*Encoder).WriteToken':
/go/pkg/mod/k8s.io/kube-openapi@v0.0.0-20221123214604-86e75ddd809a/pkg/internal/third_party/go-json-experiment/json/encode.go:395:(.text+0x207d648): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
/go/pkg/mod/k8s.io/kube-openapi@v0.0.0-20221123214604-86e75ddd809a/pkg/internal/third_party/go-json-experiment/json/encode.go:399:(.text+0x207d734): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
/go/pkg/mod/k8s.io/kube-openapi@v0.0.0-20221123214604-86e75ddd809a/pkg/internal/third_party/go-json-experiment/json/encode.go:434:(.text+0x207d8a4): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
collect2: error: ld returned 1 exit status

[liggitt]

thanks for checking... does the same issue reproduce with go1.19.5 (dropping the GOEXPERIMENT=nounified and -gcflags=all=-d=inlstaticinit=0 bits which don't apply to go1.19)?

[rfratto]

Yes, we first started to observe this with go1.19.4, and I'm able to locally reproduce the same issue using go1.19.5 as well.

[liggitt]

That's a helpful data point, thanks... sounds like this particular build / set of dependencies comes up "unlucky" as described in #57410 (comment), unrelated to changes in the go compiler to switch to the unified compiler frontend between go1.19 and go1.20

[thanm]

I am going to dup this bug against issue #58425 since it looks pretty much the same, and I've verified that the fix also eliminates the link failure here.

If you have any way to test the resulting ARM binary that would be great, thanks.

[thanm]

dup of #58425