cmd/link: slow link times with large amounts of DWARF data

[orf]

Go version

go version go1.26-devel_bffe7ad9f1 Wed Sep 17 06:43:57 2025 -0700 darwin/arm64

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN='/Users/tom/.local/share/mise/installs/go/1.25.0/bin'
GOCACHE='/Users/tom/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/tom/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/r1/myg4s_g127d6pjrb1rbry1_80000gn/T/go-build559902963=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/Users/tom/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/tom/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/tom/.local/share/mise/installs/go/1.25.0'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/tom/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/tom/.local/share/mise/installs/go/1.25.0/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.26-devel_bffe7ad9f1 Wed Sep 17 06:43:57 2025 -0700'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

We have a binary that has around ~1GB of DWARF data when built. This takes ~4 minutes, 3 of which is spent inside the linker.

What did you see happen?

When passing the -w argument to the linker to skip including DWARF data the linker runtime is reduced to ~30 seconds.

After some profiling (see attached linker-cpu.prof.gz), ~25% of linker runtime is spent inside runtime/memmove.

One of the biggest culprits seems to be WriteSym here:

go/src/cmd/link/internal/ld/outbuf.go

Lines 307 to 315 in bffe7ad

	func (out *OutBuf) WriteSym(ldr *loader.Loader, s loader.Sym) []byte {
	if !ldr.IsGeneratedSym(s) {
	P := ldr.Data(s)
	n := int64(len(P))
	pos, buf := out.writeLoc(n)
	copy(buf[pos:], P)
	out.off += n
	ldr.FreeData(s)
	return buf[pos : pos+n]

Specifically, with this binary copy() is called 5,402,448 times and moves a total of 832 MB.

What did you expect to see?

Building binaries that bring in large volumes of DWARF data should not take ~3 minutes to build.

[prattmic]

How big is the binary without DWARF? Beyond the performance issue, 1GB of DWARF seems extremely high.

[orf]

If it makes any difference, this is a test binary for a module in a very large go project. The sizes are:

• 1.1Gb with -compressdwarf=false -> ~4m 20s minutes to link
• 634Mb with -w -> ~35 seconds to link
• 520Mb with -s -> ~31 seconds to link

So my bad, it's more like ~500Mb of DWARF data.

[mknyszek]

CC @golang/compiler

[mknyszek]

@orf Is this issue present in older Go versions as well?

[orf]

It's present on 1.25, but I'm not able to easily and directly test versions older than that.

[gabyhelp]

Related Issues

• cmd/link: emitting DWARF doubles time spent in linker #26318 (closed)
• proposal: cmd/link: by default, do not write out DWARF #26074
• cmd/link: combining dwarf on darwin can be very slow #12259
• cmd/link: dwarf code badly confused by repeated local symbols #21566 (closed)

Related Code Changes

• cmd: generate DWARF for functions in compile instead of link.
• cmd/link: support DWARF compression on Darwin
• cmd: generate DWARF for functions in compile instead of link.
• cmd/link: allow enabling DWARF with -s
• [dev.link] cmd/compile, cmd/link: move away from DWARF file renumbering
• cmd/link: compress DWARF sections in ELF binaries

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[cherrymui]

~25% of linker runtime is spent inside runtime/memmove.

This is not necessarily a bad thing. In essence, this is what the linker does -- copying things from object files to the linked binary. I'd hope it be higher.

For DWARF, currently the linker does more work than just copying data. E.g. some DWARF symbols are created at link time, instead of compile time. And some of this work is sequential. So it is expected that linking a large amount of DWARF is not fast. Maybe we'll move more of the DWARF creation to the compiler and/or make them concurrent, one day.