cmd/link: relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'

[saschagrunert]

What version of Go are you using (go version)?

$ go version
go version go1.20 linux/amd64

Does this issue reproduce with the latest release?

Yes, by cross compiling kubernetes/kubernetes@b7ad179 to the linux/arm platform.

What operating system and processor architecture are you using (go env)?

We build in a custom container image registry.k8s.io/build-image/kube-cross

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/go/src/k8s.io/kubernetes/.cache/go-build"
GOENV="/go/src/k8s.io/kubernetes/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp.k8s/go-build942910395=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Running make release in the root of the Kubernetes repository. Everything gets built successfully, except the k8s.io/kubernetes/test/e2e_node.test, which fails with:

/usr/local/go/pkg/tool/linux_amd64/link: running arm-linux-gnueabihf-gcc failed: exit status 1
/tmp.k8s/go-link-865684912/go.o: in function `k8s.io/kubernetes/vendor/github.com/aws/aws-sdk-go/service/ec2.(*ModifyCapacityReservationInput).GoString':
go.go:(.text+0x207ca88): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
go.go:(.text+0x207ca94): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
collect2: error: ld returned 1 exit status

A small patch can be applied to make it fail faster for reproduction:

diff --git a/hack/make-rules/cross.sh b/hack/make-rules/cross.sh
index f8a6d0dbf5e..1a7a19ff307 100755
--- a/hack/make-rules/cross.sh
+++ b/hack/make-rules/cross.sh
@@ -27,12 +27,12 @@ source "${KUBE_ROOT}/hack/lib/init.sh"
 # NOTE: Using "${array[*]}" here is correct.  [@] becomes distinct words (in
 # bash parlance).
 
-make all WHAT="${KUBE_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_SERVER_PLATFORMS[*]}"
+#make all WHAT="${KUBE_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_SERVER_PLATFORMS[*]}"
 
-make all WHAT="${KUBE_NODE_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_NODE_PLATFORMS[*]}"
+#make all WHAT="${KUBE_NODE_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_NODE_PLATFORMS[*]}"
 
-make all WHAT="${KUBE_CLIENT_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_CLIENT_PLATFORMS[*]}"
+#make all WHAT="${KUBE_CLIENT_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_CLIENT_PLATFORMS[*]}"
 
-make all WHAT="${KUBE_TEST_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_PLATFORMS[*]}"
+#make all WHAT="${KUBE_TEST_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_PLATFORMS[*]}"
 
-make all WHAT="${KUBE_TEST_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_SERVER_PLATFORMS[*]}"
+make all WHAT="test/e2e_node/e2e_node.test" KUBE_BUILD_PLATFORMS="linux/arm"

Setting CGO_CFLAGS=-mlong-calls does not help, disabling CGO does provide a workaround.

Source code changes seem to have impact on the issue, because the build works with the next commit kubernetes/kubernetes@b0171f7

Ref: kubernetes/kubernetes#115613, kubernetes/kubernetes#115605

What did you expect to see?

That the test binary compiles correctly with CGO enabled.

What did you see instead?

The link failure.

[saschagrunert]

Copying from kubernetes/kubernetes#115605 (comment)

Reproducer with a Dockerfile, on kubernetes/kubernetes@b7ad179:

FROM debian:latest

RUN apt-get update && \
    apt-get install -y \
        gcc-arm-linux-gnueabihf \
        wget

RUN wget https://go.dev/dl/go1.20.linux-amd64.tar.gz && \
    tar xf *.tar.gz && \
    mv go /usr/local

ENV PATH=$PATH:/usr/local/go/bin
ENV GOARCH=arm
ENV CGO_ENABLED=1
ENV CC=arm-linux-gnueabihf-gcc

COPY . /work
WORKDIR /work
RUN go test -o e2e_node.test k8s.io/kubernetes/test/e2e_node

• It breaks with go 1.20 but works with go 1.19.5 on debian:latest (bullseye-20230208, arm-linux-gnueabihf-gcc 10.2.1)
• It breaks with go 1.20 but works with go 1.19.5 on debian:testing (testing-20230208, arm-linux-gnueabihf-gcc 12.2.0)

[tpaschalis]

We're also having this issue when cross-compiling the Grafana Agent on ARMv6/ARMv7. I've posted some details on #58428

[dr2chase]

@golang/compiler

[thanm]

I'll take a look.

[liggitt]

I bisected to 833367e:

git bisect start
# good: [e99f53fed98b0378c147588789b8c56b0305469b] doc: move Go 1.19 release notes to x/website
git bisect good e99f53fed98b0378c147588789b8c56b0305469b
# bad: [de4748c47c67392a57f250714509f590f68ad395] [release-branch.go1.20] go1.20
git bisect bad de4748c47c67392a57f250714509f590f68ad395
# bad: [1e4989c33674bf9570c99250a081a448c3cded65] cmd: relocate search.MatchPattern to cmd/internal/pkgpattern
git bisect bad 1e4989c33674bf9570c99250a081a448c3cded65
# bad: [00bee6d9a4c3ed6168350fc6551043ff7a1895f2] cmd/compile/internal/typebits: relax alignment check
git bisect bad 00bee6d9a4c3ed6168350fc6551043ff7a1895f2
# bad: [72a76ca1f9c195ed39e929cf768d5df5421eada1] cmd/compile: restore test/nested.go test cases
git bisect bad 72a76ca1f9c195ed39e929cf768d5df5421eada1
# good: [9e4638ad20181ba4c3b1985da48c56be121901fb] net: discard unrooted 254 byte names, not rooted ones
git bisect good 9e4638ad20181ba4c3b1985da48c56be121901fb
# good: [90466e1ddf0e4305bc56f6eac61a690704e6fab8] crypto/internal/subtle: rename to crypto/internal/alias
git bisect good 90466e1ddf0e4305bc56f6eac61a690704e6fab8
# bad: [d654117075d1dd0686aa6833b22f28275623c759] cmd/compile: add intrinsic for MulUintptr on arm64
git bisect bad d654117075d1dd0686aa6833b22f28275623c759
# good: [c82bbc0e8edbbebe47e92729e8f3f1b60d380b5b] runtime: convert timer0When/timerModifiedEarliest to atomic.Int64
git bisect good c82bbc0e8edbbebe47e92729e8f3f1b60d380b5b
# good: [9485d4c1bd871be792d03c29b7902f6ac284ed27] cmd/compile/internal/syntax: handle missing index like in go/parser
git bisect good 9485d4c1bd871be792d03c29b7902f6ac284ed27
# bad: [55ecc3a88670c14bbbfc9d6c96cf65891cc608f5] net: allow single dot in resolv.conf search statement
git bisect bad 55ecc3a88670c14bbbfc9d6c96cf65891cc608f5
# bad: [833367e98af838a2511ee7e4e19dc8f1da7b8ed7] internal/buildcfg: enable unified IR by default
git bisect bad 833367e98af838a2511ee7e4e19dc8f1da7b8ed7
# good: [17211c355f015862b635f8cbd48ffc65a1081b92] go/types, types2: consolidate testdata/spec test files
git bisect good 17211c355f015862b635f8cbd48ffc65a1081b92
# good: [0a4a57de4d0764a4291e4d8654e88b44e65d8059] cmd/compile/internal/types2: use go/types/testdata/spec tests
git bisect good 0a4a57de4d0764a4291e4d8654e88b44e65d8059
# first bad commit: [833367e98af838a2511ee7e4e19dc8f1da7b8ed7] internal/buildcfg: enable unified IR by default

Notes:

1. I had to cherry-pick 00bee6d onto each candidate commit prior to that fix before building to get around the alignment bug to make it to the failed linking step
2. linux/arm failed, linux/arm64 seemed to work

Reproducer:

# checkout of https://github.com/golang/go
pushd go
  git checkout 833367e98af838a2511ee7e4e19dc8f1da7b8ed7
  git cherry-pick 00bee6d9a4c3ed6168350fc6551043ff7a1895f2
  pushd src
    ./make.bash
  popd
popd

export GOROOT=$PWD/go
export GOPATH=$PWD/emptydir
export PATH=$PWD/go/bin:$PATH

# checkout of https://github.com/kubernetes/kubernetes/
pushd kubernetes
  git checkout b7ad17978eaba508c560f81bab2fb9d0b256e469
  GOARCH=arm CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc go test -c k8s.io/kubernetes/test/e2e_node
popd

Output is:

# k8s.io/kubernetes/test/e2e_node.test
/usr/local/google/home/liggitt/tmp/arm/go/pkg/tool/linux_amd64/link: running arm-linux-gnueabihf-gcc failed: exit status 1
/tmp/go-link-3447838833/go.o: in function `github.com/aws/aws-sdk-go/service/ec2.(*DescribeScheduledInstancesInput).GoString':
./<autogenerated>:1:(.text+0x207e440): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
/tmp/go-link-3447838833/go.o: in function `github.com/aws/aws-sdk-go/service/ec2.(*DescribeScheduledInstancesInput).GoString':
/usr/local/google/home/liggitt/tmp/arm/kubernetes/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:94642:(.text+0x207e44c): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
/tmp/go-link-3447838833/go.o: in function `github.com/aws/aws-sdk-go/service/ec2.(*DescribeScheduledInstancesInput).String':
./<autogenerated>:1:(.text+0x207e4f0): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
/tmp/go-link-3447838833/go.o: in function `github.com/aws/aws-sdk-go/service/ec2.(*ScheduledInstance).GoString':
./<autogenerated>:1:(.text+0x207e598): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
/tmp/go-link-3447838833/go.o: in function `github.com/aws/aws-sdk-go/service/ec2.(*ScheduledInstance).GoString':
/usr/local/google/home/liggitt/tmp/arm/kubernetes/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:153021:(.text+0x207e5a4): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
collect2: error: ld returned 1 exit status

That commit defaulted GOEXPERIMENT=Unified on. Starting to run the bisection again with that experiment on in the invocation

[liggitt]

Building go1.19.5 with GOEXPERIMENT=Unified reproduces the relocation truncated to fit: R_ARM_CALL against 'runtime.duffcopy' issue, so it looks like the unified build regresses on arm

[liggitt]

cc @mdempsky

[liggitt]

until this issue is fixed, I was able to build with GOEXPERIMENT=nounified to work around this issue and -gcflags=all=-d=inlstaticinit=0 to work around #58339

[thanm]

@liggitt thanks for the bisect.

In this case my guess would be that the bug is not actually due to the introduction of unified IR, but rather that the switch to unified IR makes the binary in question a bit bigger, which triggers the need for trampoline insertion.

I am testing a CL with a fix, I'll post it shortly.

[gopherbot]

Change https://go.dev/cl/467715 mentions this issue: cmd/link/internal/ld: fix text section splitting for ARM

[thanm]

@liggitt if you are able to test with my CL to see if the tests run cleanly on ARM that would be great.

I verified that with the CL I can link the k8s.io/kubernetes/test/e2e_node test, but my ARM test machine doesn't seem to be configured properly for me to run it (no "modprobe", , or something like that).

[liggitt]

I don't actually have an arm test machine, all I could do was reproduce / verify the cross-compile/link worked. I'll see if I can track down an env to run the resulting binary.

I'd be curious if the CL also resolves #58428 ... the reporters of that issue might have an env to exercise their resulting binary as well

[thanm]

OK thanks. Yes, I verified that the fix also takes care of #58428. I am planning on duping that bug against this one. I'll on that issue about testing.

[rfratto]

👋 We've tested your CL against the resulting binary for #58428 and verified that everything still works. Thank you!

[liggitt]

Thanks for the quick turnaround. Could this be backported to 1.20, since it resolves a compiler issue that doesn't have a workaround (#58428)?

[thanm]

I will indeed look into backports.

[thanm]

Hmm, gold/arm.cc:4079 is an assert related to call stubs, so this is almost certainly the same problem (in some new form). Could be that we need to extend the fix from https://go-review.git.corp.google.com/c/go/+/474620 to 32-bit arm (currently it is for 64-bit arm).

[cherrymui]

We can still apply the previous fix. Sometimes the C linker just doesn't like the large text sections, even if we got the trampolines set up already. So splitting the text section may still be a good thing to do.

[thanm]

I spent a little while debugging this using grafana-agent, e.g.

$ git clone https://github.com/grafana/agent.git
$ cd agent
$ git checkout 2eb10ae840faf565f8eb063a79a8f1d826fb9483
$ cat t.sh
#!/bin/sh
set -x
set -e
rm -f grafana-agent.exe
rm -rf tmpdir
mkdir tmpdir
GOOS=linux \
  CC=/usr/bin/arm-linux-gnueabihf-gcc \
  GOARCH=arm \
  GOARM=7 \
  CGO_ENABLED=1 \
  CGO_CFLAGS=" -mlong-calls" \
  go build  -ldflags="-v -tmpdir=tmpdir" -tags "netgo " \
    -x -o /tmp/grafana-agent.exe  ./cmd/grafana-agent
$

The error I get is a little different from the one reported in 58425#issuecomment-1464832903; I get:

tmpdir/go.o: in function `k8s.io/api/storage/v1.(*VolumeError).Unmarshal':
/ssd2/go1/pkg/mod/k8s.io/api@v0.25.4/storage/v1/generated.pb.go:5158:(.text+0x207bc74): relocation truncated to fit: R_ARM_CALL against `runtime.morestack_noctxt'

Target of the relocation is not a runtime.*duff* routine, so this doesn't relate to the previous bug in which we had a call relocation with non-zero addend.

I looked at the relocation in question, and from my calculations the calculated displacement is exactly at the lower limit of the relocation in question (e.g. the minimum value represented by a signed 24-bit quantity, or -0x800000). Source of the relocation (this is from objdump -dr):

 207bc70:      	mov	r3, lr
 207bc74:      	bl	0x207bc74 <k8s.io/api/storage/v1.(*VolumeError).Unmarshal+0x6cc> @ imm = #-8
			0207bc74:  R_ARM_CALL	runtime.morestack_noctxt

and target of the relocation is at 0x0007bc74 with offset of 1740, so this computation here yields exactly -0x800000.

Bottom line is that this is a disagreement between the Go linker's calculation of what will reach and the external linker's calculation.

For this line here in the Go linker

	if t > 0x7fffff || t < -0x800000 || ... 

if I change t < -0x800000 to t <= -0x800000 the program links ok.

So although the symptom is the same, this looks like a slightly different bug.

[thanm]

Also tried the "t <= -0x800000" change and that seems to fix the problem with the Gold linker as well. I will send a CL.

[gopherbot]

Change https://go.dev/cl/475957 mentions this issue: cmd/link/internal/arm: fix off-by-1 in trampoline reachability computation

[liggitt]

thanks, can that fix be taken back to 1.19 / 1.20?

[thanm]

Yes, I've kicked off the process to back-port the second fix (issue to follow is #59034).

[gopherbot]

Change https://go.dev/cl/476935 mentions this issue: cmd/link/internal/arm: fix off-by-1 in trampoline reachability computation

[gopherbot]

Change https://go.dev/cl/476936 mentions this issue: cmd/link/internal/arm: fix off-by-1 in trampoline reachability computation

[rfratto]

We just updated to go1.20.3, and our 32-bit ARM builds seem to be working again 🎉 Thanks @thanm!

[rfratto]

@thanm Unfortunately I spoke too soon. grafana/agent#3485 just ran into the issue again after removing some code, with two different errors for two different commands (both ARMv6):

# github.com/grafana/agent/cmd/grafana-agent-flow
/usr/local/go/pkg/tool/linux_amd64/link: running viceroycc failed: exit status 1
/tmp/go-link-4002740392/go.o: in function `github.com/weaveworks/common/middleware.Log.Wrap.func1':
/go/pkg/mod/github.com/weaveworks/common@v0.0.0-20221201103051-7c2720a9024d/middleware/response.go:45:(.text+0x207dab0): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'

# github.com/grafana/agent/cmd/grafana-agentctl
/usr/local/go/pkg/tool/linux_amd64/link: running viceroycc failed: exit status 1
/tmp/go-link-1646694243/go.o: in function `google.golang.org/api/compute/v1.(*RegionSecurityPoliciesListCall).doRequest':
/go/pkg/mod/google.golang.org/api@v0.109.0/compute/v1/compute-gen.go:153224:(.text+0x207d7f4): relocation truncated to fit: R_ARM_CALL against `runtime.gcWriteBarrier'

This can be validated by checking out 0369aedde7d227b221eaf97c6b489a433f21ba5d and building ./cmd/grafana-agentctl and ./cmd/grafana-agent-flow with CGO_ENABLED=1 GOOS=linux GOARCH=arm GOARM=6

[thanm]

Thanks for the update. This bug seems to be the "gift that keeps giving", apparently. I will take another look.

[mknyszek]

Hey @thanm, doing a sweep of the Go 1.21 milestone. Any updates here? Should this go to Backlog? Thanks.

[thanm]

Should still be in 1.21 I think (just got a bit lost in the sauce). I will take another look.

[kminehart]

This strangely started showing up in Grafana. We bisected to a seemingly benign commit, though it is consistently failing in a function in a specific dependency. Based on what I'm seeing in this thread, the dependency is unrelated.

We are seeing this on 1.21.1, but we are not seeing this error on 1.21.0 or 1.20.8.

CC=/toolchain/arm-linux-musleabihf-cross/bin/arm-linux-musleabihf-gcc
CGO_CFLAGS="-D_LARGEFILE64_SOURCE"
CGO_ENABLED=1
CXX=/toolchain/arm-linux-musleabihf-cross/bin/arm-linux-musleabihf-cpp
GOARCH=arm
GOARM=7
GOOS=linux

go build -ldflags -w -s -linkmode=external -extldflags=-static -X 'main.version=10.2.0-pre' -X 'main.commit=40c7aac087' -X 'main.buildstamp=1694554530' -X 'main.buildBranch=main'  -o bin/linux/arm/v7/grafana-cli -tags netgo,osusergo -trimpath ./pkg/cmd/grafana

/usr/local/go/pkg/tool/linux_amd64/link: running /toolchain/arm-linux-musleabihf-cross/bin/arm-linux-musleabihf-gcc failed: exit status 1
/tmp/go-link-2970699141/go.o: in function `github.com/rueian/rueidis/internal/cmds.(*ClientTrackingBcast).Optout':
go.go:(.text+0x2081d14): relocation truncated to fit: R_ARM_CALL against `runtime.gcWriteBarrier2'
collect2: error: ld returned 1 exit status

[cherrymui]

This failure is very sensitive to binary size and and code layout. A seemingly unrelated change may change the code layout just slightly to go over or under the limit.