cmd/link: Oversize object file (example uses generics) causes confusing "slice bounds out of range" panic

[djedward]

Go version

go version go1.21.8 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/dj/.cache/go-build'
GOENV='/home/dj/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/dj/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/dj/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.8'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/dj/gosize/code/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2460092799=/tmp/go-build -gno-record-gcc-switches'

What did you do?

We updated our AWS SDK v1 version and existing code started failing. I distilled it down to this minimal case. The issue happens when the generated object in the GOCACHE goes above 4GB. The goobj/objfile Header appears to be using a uint32, which seems like an overflow might be happening when calculating the offsets. I dumped the headers from one of the objects while investigating (not the minimal case above, so offsets are a little different):

BlkAutolib: 3225526148
BlkPkgIdx: 3225526596
BlkFile: 3225526884
BlkSymdef: 3225528524
BlkHashed64def: 3225727751
BlkHasheddef: 3225729032
BlkNonpkgdef: 3226325537
BlkNonpkgref: 3226479026
BlkRefFlags: 3226587890
BlkHash64: 3226587890
BlkHash: 3226588378
BlkRelocIdx: 3227042858
BlkAuxIdx: 3227223910
BlkDataIdx: 3227404962
BlkReloc: 3227586014
BlkAux: 3228926500
BlkData: 3229305472
BlkRefName: 10123514
BlkEnd: 10148874

I tested it in a few other versions:
1.22.1 - compiles
1.22.0 - compiles
1.22rc2 - compiles
1.22rc1 - fails for same reason

What did you see happen?

panic: runtime error: slice bounds out of range [::3237855945] with capacity 26478551

goroutine 1 [running]:
cmd/internal/goobj.(*Reader).BytesAt(...)
	cmd/internal/goobj/objfile.go:622
cmd/internal/goobj.(*Reader).uint32At(...)
	cmd/internal/goobj/objfile.go:635
cmd/internal/goobj.(*Reader).StringRef(0x0?, 0x1be438?)
	cmd/internal/goobj/objfile.go:675 +0xf8
cmd/internal/goobj.(*Reader).Autolib(0xc000014120)
	cmd/internal/goobj/objfile.go:688 +0x71
cmd/link/internal/loader.(*Loader).Preload(0xc0000ec000, 0xa, 0xc000028020, 0xc0001c2000, 0xc0000cc410, 0x46f725?)
	cmd/link/internal/loader/loader.go:2147 +0x395
cmd/link/internal/ld.ldobj(0xc00019e000, 0xc000028020, 0xc0001c2000, 0x194086c, {0xc00001a300, 0x2c}, {0x7ffcd10911bc, 0x24})
	cmd/link/internal/ld/lib.go:2294 +0xa8f
cmd/link/internal/ld.loadobjfile(0xc00019e000, 0xc0001c2000)
	cmd/link/internal/ld/lib.go:1136 +0x7f3
cmd/link/internal/ld.(*Link).loadlib(0xc00019e000)
	cmd/link/internal/ld/lib.go:544 +0x214
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0xc000012439, 0x1, 0x1}, ...})
	cmd/link/internal/ld/main.go:284 +0x104b
main.main()
	cmd/link/main.go:72 +0xdfb

What did you expect to see?

A successful build

[cherrymui]

Currently there is no plan to support an object larger than 4 GiB. Try refactoring the code or splitting to multiple objects. We can emit a better error, though.

[djedward]

Sorry for not being clear, I'm not asking for >4GB support. Having a better error message would suffice. Especially because we are unblocked by just using Go 1.22 which generates a much smaller object. I didn't analyze the objects from 1.22, but the largest object in the cache is ~75MB for the same sample program.

I mainly opened the issue because the behavior is so vastly different between 1.21 and 1.22. The suspected overflow seems to result in strange offsets. The end offset in the object says it ends at about 10MB, but the previous offsets are in the 3GB range. I couldn't find another object in the cache that seems to have an end offset before the other offsets. So my thought was that if the overflow issue was addressed, it might actually produce a much smaller object file and be able to compile. But this is my first time looking at any of this code (I freely admit being a complete noob 😄), so it might be completely acceptable that those offsets are after the end offset.