x/tools/gopls: incorporate pgo into release process

[findleyr]

Reminder issue: we should use pgo with gopls.

We can either calibrate using our benchmarks, or by collecting/merging real editing sessions.

[qiulaidongfeng]

I have looked into optimizing gopls with pgo, but there are few benchmarks in gopls, so it is difficult to measure the performance gains from pgo.
I'm curious how to measure the optimization that pgo brings to gopls?

[findleyr]

@qiulaidongfeng there are benchmarks for performance critical operations, across a variety of open-source repositories, defined in the x/tools/gopls/internal/regtest/bench package. These produce the dashboard I linked above, which we monitor to detect performance regressions.

Can you explain why these are inadequate for evaluating pgo? (serious question: perhaps I'm missing something).

[qiulaidongfeng]

The command I use is go test -run=ssss -bench=. -benchtime=500ms ./...
The output is this
? golang.org/x/tools/gopls [no test files]

D:\file\gofile\gogit\tools\gopls>go test -run=ssss -bench=. -benchtime=500ms ./...
? golang.org/x/tools/gopls [no test files]
? golang.org/x/tools/gopls/api-diff [no test files]
PASS
ok golang.org/x/tools/gopls/doc 0.216s
? golang.org/x/tools/gopls/integration/govim [no test files]
PASS
ok golang.org/x/tools/gopls/internal/astutil 0.402s
PASS
ok golang.org/x/tools/gopls/internal/bug 0.380s
PASS
ok golang.org/x/tools/gopls/internal/cmd 0.495s
? golang.org/x/tools/gopls/internal/file [no test files]
PASS
ok golang.org/x/tools/gopls/internal/goversion 0.190s
PASS
ok golang.org/x/tools/gopls/internal/hooks 0.242s
? golang.org/x/tools/gopls/internal/immutable [no test files]
PASS
ok golang.org/x/tools/gopls/internal/lsp 0.222s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/deprecated 0.337s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/embeddirective 0.337s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/fillreturns 0.335s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/fillstruct 0.320s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/infertypeargs 0.332s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/nonewvars 0.337s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/noresultvalues 0.334s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/simplifycompositelit 0.325s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/simplifyrange 0.333s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/simplifyslice 0.331s
? golang.org/x/tools/gopls/internal/lsp/analysis/stubmethods [no test files]
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/undeclaredname 0.323s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/unusedparams 0.334s
? golang.org/x/tools/gopls/internal/lsp/analysis/unusedparams/cmd [no test files]
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/unusedvariable 0.332s
PASS
ok golang.org/x/tools/gopls/internal/lsp/analysis/useany 0.332s
? golang.org/x/tools/gopls/internal/lsp/browser [no test files]
PASS
ok golang.org/x/tools/gopls/internal/lsp/cache 0.236s
PASS
ok golang.org/x/tools/gopls/internal/lsp/command 0.194s
? golang.org/x/tools/gopls/internal/lsp/command/commandmeta [no test files]
? golang.org/x/tools/gopls/internal/lsp/command/gen [no test files]
PASS
ok golang.org/x/tools/gopls/internal/lsp/debug 0.337s
? golang.org/x/tools/gopls/internal/lsp/debug/log [no test files]
PASS
ok golang.org/x/tools/gopls/internal/lsp/fake 0.303s
goos: windows
goarch: amd64
pkg: golang.org/x/tools/gopls/internal/lsp/filecache
cpu: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics
BenchmarkUncontendedGet-16 2324030 253.4 ns/op 32333.17 MB/s
BenchmarkUncontendedSet-16 5 117142780 ns/op 69.93 MB/s
PASS
ok golang.org/x/tools/gopls/internal/lsp/filecache 2.610s
PASS
ok golang.org/x/tools/gopls/internal/lsp/frob 0.226s
PASS
ok golang.org/x/tools/gopls/internal/lsp/glob 0.231s
PASS
ok golang.org/x/tools/gopls/internal/lsp/lru 0.303s
PASS
ok golang.org/x/tools/gopls/internal/lsp/lsprpc 0.418s
? golang.org/x/tools/gopls/internal/lsp/mod [no test files]
PASS
ok golang.org/x/tools/gopls/internal/lsp/progress 0.105s
PASS
ok golang.org/x/tools/gopls/internal/lsp/protocol 0.276s
PASS
ok golang.org/x/tools/gopls/internal/lsp/protocol/generate 0.236s
PASS
ok golang.org/x/tools/gopls/internal/lsp/regtest 0.338s
PASS
ok golang.org/x/tools/gopls/internal/lsp/safetoken 0.302s
PASS
ok golang.org/x/tools/gopls/internal/lsp/snippet 0.198s
PASS
ok golang.org/x/tools/gopls/internal/lsp/source 0.185s
PASS
ok golang.org/x/tools/gopls/internal/lsp/source/completion 0.175s
? golang.org/x/tools/gopls/internal/lsp/source/methodsets [no test files]
goos: windows
goarch: amd64
pkg: golang.org/x/tools/gopls/internal/lsp/source/typerefs
cpu: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics
BenchmarkBuildPackageGraph-16 4 171149075 ns/op
--- BENCH: BenchmarkBuildPackageGraph-16
pkgrefs_test.go:257: loaded 810 packages in 1.240107s
pkgrefs_test.go:257: loaded 810 packages in 1.2555198s
pkgrefs_test.go:257: loaded 810 packages in 1.2275294s
pkgrefs_test.go:257: loaded 810 packages in 1.2527227s
PASS
ok golang.org/x/tools/gopls/internal/lsp/source/typerefs 6.924s
? golang.org/x/tools/gopls/internal/lsp/source/xrefs [no test files]
PASS
ok golang.org/x/tools/gopls/internal/lsp/template 0.227s
? golang.org/x/tools/gopls/internal/lsp/tests [no test files]
PASS
ok golang.org/x/tools/gopls/internal/lsp/tests/compare 0.203s
? golang.org/x/tools/gopls/internal/lsp/work [no test files]
? golang.org/x/tools/gopls/internal/pathutil [no test files]
PASS
ok golang.org/x/tools/gopls/internal/persistent 0.098s
2023/11/21 21:33:39 cloning https://github.com/googleapis/google-cloud-go.git@07da765765218debf83148cc7ed8a36d6e8921d5 into C:\Users\26454\AppData\Local\Temp\gopls-bench2084362444\google-cloud-go
2023/11/21 21:33:39 checking out https://github.com/googleapis/google-cloud-go.git: exec: "/bin/sh": executable file not found in %PATH%
exit status 1
FAIL golang.org/x/tools/gopls/internal/regtest/bench 0.491s
PASS
ok golang.org/x/tools/gopls/internal/regtest/codelens 0.493s
PASS
ok golang.org/x/tools/gopls/internal/regtest/completion 0.488s
PASS
ok golang.org/x/tools/gopls/internal/regtest/debug 0.493s
PASS
ok golang.org/x/tools/gopls/internal/regtest/diagnostics 0.487s
PASS
ok golang.org/x/tools/gopls/internal/regtest/inlayhints 0.497s
PASS
ok golang.org/x/tools/gopls/internal/regtest/marker 0.489s
PASS
ok golang.org/x/tools/gopls/internal/regtest/misc 0.510s
PASS
ok golang.org/x/tools/gopls/internal/regtest/modfile 0.488s
PASS
ok golang.org/x/tools/gopls/internal/regtest/template 0.497s
PASS
ok golang.org/x/tools/gopls/internal/regtest/watch 0.496s
PASS
ok golang.org/x/tools/gopls/internal/regtest/workspace 0.525s
PASS
ok golang.org/x/tools/gopls/internal/settings 0.389s
PASS
ok golang.org/x/tools/gopls/internal/telemetry 0.570s
? golang.org/x/tools/gopls/internal/telemetry/cmd/stacks [no test files]
? golang.org/x/tools/gopls/internal/vulncheck [no test files]
? golang.org/x/tools/gopls/internal/vulncheck/govulncheck [no test files]
? golang.org/x/tools/gopls/internal/vulncheck/osv [no test files]
? golang.org/x/tools/gopls/internal/vulncheck/scan [no test files]
PASS
ok golang.org/x/tools/gopls/internal/vulncheck/semver 0.230s
PASS
ok golang.org/x/tools/gopls/internal/vulncheck/vulntest 0.340s
? golang.org/x/tools/gopls/release [no test files]
FAIL

[qiulaidongfeng]

@findleyr
I didn't see before x/tools/gopls/internal/regtest/bench, because I use of pgo from go1.21 released just a few months.
I have been using go test-bench =. Maybe I fell into a mindset, sorry to make you confused.

[findleyr]

Ah, those benchmarks are failing because they use /bin/sh (the perf builders are linux). We can and should fix this.