cmd/compile: increased compilation time for a parser package generated with goyacc

[yuzefovich]

Go version

go version go1.25.5 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/yahor_cockroachlabs_com/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/yahor_cockroachlabs_com/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build471937613=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/home/yahor_cockroachlabs_com/go/src/github.com/cockroachdb/cockroach/go.mod'
GOMODCACHE='/home/yahor_cockroachlabs_com/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/yahor_cockroachlabs_com/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/yahor_cockroachlabs_com/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.5.linux-amd64'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/yahor_cockroachlabs_com/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/yahor_cockroachlabs_com/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.5.linux-amd64/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.5'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

See below.

What did you see happen?

Over in CockroachDB we have a sql/parser package that mostly consists of the generated code from goyacc sql.go.zip. Over the last few years we noticed that the compilation time of this package has increased significantly with some new major Go releases. Part of it could be attributed to the growth of the package itself, but I believe most of the increase is due to some compiler work (which we do appreciate!).

Here are statistics I've gathered. Each bullet point compares compiling the same generated code but using different Go versions (the code across bullet points is different):

• generated parser/sql.go file: 46954 lines
  • go 1.21.5: 18.8s
  • go 1.22.0: 22.9s
• generated parser/sql.go file: 48952 lines
  • go 1.22.5: 26.5s
  • go 1.23.2: 26.8s
• generated parser/sql.go file: 52343 lines
  • go 1.23.12: 38.3s
  • go 1.25.3: 34.6s (we by-passed go 1.24)
• generated parser/sql.go file: 52642 lines (tip of master)
  • Go 1.25.5: 34.9s

Examining the debug-trace trace.zip we see that vast majority is spent building the package in a single thread

Main observation

So overall, between 1.21.5 and 1.25.5 versions, while the code size increased by 12%, the build time increased by 86%. (The code generated by goyacc shouldn't have changed in a significant manner.)

What did you expect to see?

My hypothesis is that the shape of the generated parser code somehow doesn't play nicely with recent compiler improvements. I'd be curious to see whether improving compile time of such code is on the radar of the team. (Setting up the environment to build CockroachDB is a bit involved, so if there is interest, I could probably come up with a sample code as the benchmark.)

(Note that CockroachDB uses a fork of golang/go with a handful of patches on top that shouldn't impact the compile time.)

[gabyhelp]

Related Issues

• cmd/compile: consider speeding up compile of large ast package in microsoft/typescript-go #73044
• cmd/compile: long build time (45 seconds) for a small package (1500 source lines of code) #65097 (closed)
• cmd/compile: compileSSA takes a long time on very large (generated) functions #30077
• cmd/compile: 8x increase in compile time from go1.6 to go.tip #15520 (closed)
• cmd/compile: slow escape analysis in large package in the typescript compiler #72815 (closed)
• cmd/compile: compilation of a package >100x slower with 1.20rc3 #57959 (closed)
• cmd/compile: compile time increased greatly in 1.25rc1 with commit ed24bb4e60 #74615 (closed)
• cmd/compile/internal/escape: performance issue #67313 (closed)
• cmd/compile: slow compile/build compared to Go 1.4 #13646 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[Jorropo]

Dumb question how can I debug this ?

hugo@rikus /tmp> git clone --depth 1 https://github.com/cockroachdb/cockroach && cd cockroach/pkg/sql/parser && go build -gcflags=-cpuprofile=/tmp/a.pprof .
Cloning into 'cockroach'...
remote: Enumerating objects: 21789, done.
remote: Counting objects: 100% (21789/21789), done.
remote: Compressing objects: 100% (16384/16384), done.
remote: Total 21789 (delta 5011), reused 19208 (delta 4737), pack-reused 0 (from 0)
Receiving objects: 100% (21789/21789), 248.14 MiB | 16.64 MiB/s, done.
Resolving deltas: 100% (5011/5011), done.
Updating files: 100% (19320/19320), done.
# github.com/cockroachdb/cockroach/pkg/geo/geopb
../../geo/geopb/config.go:10:11: undefined: Config
../../geo/geopb/config.go:16:11: undefined: Config
../../geo/geopb/config.go:22:11: undefined: Config
../../geo/geopb/geopb.go:14:10: undefined: SpatialObject
../../geo/geopb/geopb.go:21:10: undefined: SpatialObject
../../geo/geopb/geopb.go:35:9: undefined: ShapeType
../../geo/geopb/geopb.go:37:7: undefined: ShapeType_Unset
../../geo/geopb/geopb.go:38:10: undefined: ShapeType_Unset
../../geo/geopb/geopb.go:39:7: undefined: ShapeType_Point
../../geo/geopb/types.go:28:9: undefined: ShapeType
../../geo/geopb/geopb.go:39:7: too many errors
# github.com/cockroachdb/cockroach/pkg/sql/lexbase
../lexbase/predicates.go:60:17: undefined: reservedKeywords

[...] many cut out errors

[yuzefovich]

We use Bazel to build. We also have dev wrapper on top of bazel to simplify some things. Once you have everything ready (i.e. ./dev build short works), you can run ./dev gen go to generate all necessary code, and at that point you can switch to using your local go tooling to compile the packages and debug.

[yuzefovich]

Attaching the CPU profile from compiling the parser package pprof.zip.

[cherrymui]

Looking at the CPU profile, most of the time is spent in the register allocator, specifically cmd/compile/internal/ssa.(*regAllocState).computeLive. Given that the trace shows a single thread, it is probably all spent for a single giant function, perhaps *sqlParserImpl.Parse, which is about 20000 lines with thousands of switch cases. Functions like this probably triggers some pathological nonlinear behavior of the register allocator.

cc @golang/compiler

[cherrymui]

In that function, a large number of switch cases don't return. They run to the end of the switch statement, which jumps to label sqlstack. This creates a control flow for a block with a huge number of predecessors, and it is in a loop (as it goto the top). I can imagine this type of control flow doesn't play well with the register allocator...

[randall77]

Can you try 1.26rc1? There was an optimization of this part of regalloc.
https://go-review.googlesource.com/c/go/+/695255
@DanielMorsing

[yuzefovich]

Thanks for prompt replies! Here is the profile when using go1.26rc1@latest (v0.0.0-20251216200105-f72f05852d7a) pprof.zip. Seems like the build time improved a little.

[DanielMorsing]

The new profile spending most of it in ComputeDesired sounds about right, it was the new slowest bit of the liveness analysis when I profiled the pass after my work, and because the analysis isn't in SSA form, there wasn't a straightforward way of speeding it up.

The desired calculation uses a linear probe, accompanied with a comment that it should be fine since the working set is small. Maybe that assumption is no longer true and we need to make it a bit more clever?