cmd/compile: worse performance when built with pgo

[crazycs520]

Go version

go version go1.23.5 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/transaction/.cache/go-build'
GOENV='/home/transaction/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/transaction/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/transaction/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/transaction/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.5.linux-amd64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/transaction/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.5.linux-amd64/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.5'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/transaction/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/transaction/tidb/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build338251587=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Building with PGO, but result in performance regressions.

The build flag is:

CGO_ENABLED=1  GO111MODULE=on go build -tags codes -pgo=default.pgo  -ldflags '-X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-alpha-206-gaccc969cd5" -X "github.com/pingcap/tidb/pkg/util/versioninfo.TiDBBuildTS=2025-01-23 06:01:16" -X "github.com/pingcap/tidb/pkg/util/versioninfo.TiDBGitHash=accc969cd5e95a992368249fcf86f11887e7b11a" -X "github.com/pingcap/tidb/pkg/util/versioninfo.TiDBGitBranch=pgo-read-only-dev5" -X "github.com/pingcap/tidb/pkg/util/versioninfo.TiDBEdition=Community" ' -o bin/tidb-server ./cmd/tidb-server

Then start the TiDB and run sysbench test, following is the test result, the QPS of building with PGO is lower.

workload	thread	Version	QPS
batch_get	64	without-pgo	23165
batch_get	64	with-pgo	21077

What did you see happen?

The performance of build with PGO is worse because newstack appeared during execution.

profile.zip

CPU profile of building without PGO:

CPU profile of building with PGO:

What did you expect to see?

no newstack during execution.

Is there any way to avoid newstack during execution, such as set stack size when creating goroutine?

[gabyhelp]

Related Issues

• cmd/compile: pgo after CL 449477 cause worse performance on benchmark #68974 (closed)
• cmd/compile: pgo causes worse performance on some benchmarks #68815
• cmd/compile: Building a tool chain using PGO does not achieve the 2-7% improvement claimed by blogs #63407 (closed)
• cmd/compile: poor register allocation with PGO #58298

Related Code Changes

• cmd/compile: build compiler with PGO

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

how did you measure performance?

[crazycs520]

how did you measure performance?

I do performance test by sysbench, following is the test result, the QPS of building with PGO is lower.

workload	thread	Version	QPS
batch_get	64	without-pgo	23165
batch_get	64	with-pgo	21077

[mknyszek]

In triage we think that maybe additional inlining made stack frames just big enough to make some short-lived goroutines have to grow the stack more often.

I would've hoped the adaptive stack presizing we do would've smoothed some of this over, but the difference is fairly large.