Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: worse performance when built with pgo #71400

Open
crazycs520 opened this issue Jan 23, 2025 · 4 comments
Open

cmd/compile: worse performance when built with pgo #71400

crazycs520 opened this issue Jan 23, 2025 · 4 comments
Assignees
Labels
BugReport Issues describing a possible bug in the Go implementation. compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance
Milestone

Comments

@crazycs520
Copy link

crazycs520 commented Jan 23, 2025

Go version

go version go1.23.5 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/transaction/.cache/go-build'
GOENV='/home/transaction/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/transaction/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/transaction/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/transaction/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.5.linux-amd64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/transaction/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.5.linux-amd64/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.5'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/transaction/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/transaction/tidb/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build338251587=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Building with PGO, but result in performance regressions.

The build flag is:

CGO_ENABLED=1  GO111MODULE=on go build -tags codes -pgo=default.pgo  -ldflags '-X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-alpha-206-gaccc969cd5" -X "github.com/pingcap/tidb/pkg/util/versioninfo.TiDBBuildTS=2025-01-23 06:01:16" -X "github.com/pingcap/tidb/pkg/util/versioninfo.TiDBGitHash=accc969cd5e95a992368249fcf86f11887e7b11a" -X "github.com/pingcap/tidb/pkg/util/versioninfo.TiDBGitBranch=pgo-read-only-dev5" -X "github.com/pingcap/tidb/pkg/util/versioninfo.TiDBEdition=Community" ' -o bin/tidb-server ./cmd/tidb-server

Then start the TiDB and run sysbench test, following is the test result, the QPS of building with PGO is lower.

workload thread Version QPS
batch_get 64 without-pgo 23165
batch_get 64 with-pgo 21077

What did you see happen?

The performance of build with PGO is worse because newstack appeared during execution.

profile.zip

CPU profile of building without PGO:

Image

CPU profile of building with PGO:

Image

What did you expect to see?

no newstack during execution.

Is there any way to avoid newstack during execution, such as set stack size when creating goroutine?

@gabyhelp gabyhelp added the BugReport Issues describing a possible bug in the Go implementation. label Jan 23, 2025
@seankhliao seankhliao changed the title building with PGO may result in performance regressions cmd/compile: worse performance when built with pgo Jan 23, 2025
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jan 23, 2025
@seankhliao
Copy link
Member

how did you measure performance?

@crazycs520
Copy link
Author

how did you measure performance?

I do performance test by sysbench, following is the test result, the QPS of building with PGO is lower.

workload thread Version QPS
batch_get 64 without-pgo 23165
batch_get 64 with-pgo 21077

@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 27, 2025
@cagedmantis cagedmantis added this to the Backlog milestone Jan 27, 2025
@mknyszek
Copy link
Contributor

In triage we think that maybe additional inlining made stack frames just big enough to make some short-lived goroutines have to grow the stack more often.

I would've hoped the adaptive stack presizing we do would've smoothed some of this over, but the difference is fairly large.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BugReport Issues describing a possible bug in the Go implementation. compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance
Projects
Development

No branches or pull requests

7 participants