x/tools/gopls: The completion feature of gopls has high latency when dealing with very large packages.

[pand5a]

gopls version

golang.org/x/tools/gopls v0.16.2

go env

GO111MODULE='auto'
GOARCH='amd64'
GOBIN=''
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH=''
GOPRIVATE=''
GOROOT='/usr/lib/go'
GOSUMDB='off'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.1'
GODEBUG=''
GOTELEMETRY='local'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1211819663=/tmp/go-build -gno-record-gcc-switches'

What did you do?

The completion feature of gopls has high latency when dealing with very large packages. Below are the completion latencies observed in the sql package of the cockroachdb source code.

The code statistics for the sql package in cockroachdb are as follows:

cloc cockroach/pkg/sql             
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Go                            2974         148944         228093        1355540
-------------------------------------------------------------------------------
SUM:                          3504         152684         236151        1424410
-------------------------------------------------------------------------------

The distribution of the main function latencies in gopls completion is as follows:

completion.go:54: elapsed 489.202836ms,        Completion entry, 
completion.go:484: elapsed 477.769576ms,            NarrowestPackageForFile, 
snapshot.go:87: elapsed 477.709921ms,                   TypeCheck, 
check.go:415: elapsed 37.059356ms,                          getPackageHandles,
check.go:422: elapsed  51.8002ms,                           getImportGraph, 
check.go:606: elapsed   194.72µs,                           getImportPackage
check.go:629: elapsed 387.29934ms,                          checkPackage, 
check.go:1614: elapsed 385.05347ms,                             check.Files

The most time-consuming part is ** check.go:1614: elapsed 385.05347ms, check.Files, 288 **, which takes around 385 milliseconds. Adding other operations, the total latency reaches 489.202836ms. This results in noticeable lag during use.

What did you see happen?

During each completion, a TypeCheck is performed on the package containing the currently modified file, and this process is very time-consuming for large packages.

What did you expect to see?

Is it possible to only re-parse the current file during completion, rather than performing a TypeCheck on the entire package? This would allow completion to be completed in milliseconds even for large packages.

Editor and settings

nvim

Logs

No response

[gabyhelp]

Related Issues and Documentation

• x/tools/gopls: >45s for first completion of unimported package #36222 (closed)
• x/tools/gopls: gopls falls behind user input in large repository #45089 (closed)
• x/tools/gopls: cannot handle large files #33372 (closed)
• x/tools/gopls: slow performance #43259 (closed)
• x/tools/gopls: completion often uses all of 1s time budget #68063
• x/tools/gopls: completions taking 20x as long since https://github.com/golang/tools/commit/41e4e5654988 #62665 (closed)
• x/tools/gopls: don't apply completion budget to shallow completions #41434 (closed)
• go/types: slow type checking with thousands of methods on a type #66699
• x/tools/gopls: slow when typing imports #34414 (closed)
• x/tools/gopls: cannot choose the "right" value for completionBudget #63459

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[pand5a]

Normal speed: in cockroachdb pkg/cmd/dev/main.go

Slow speed: in cockroachdb pkg/sql package

[findleyr]

Thanks. There is currently a hard speed-limit for this completion, based on the fact that we cannot start streaming partial results (at least for VS Code: microsoft/vscode#105870), and therefore must await type checking of this very large package.

However, I'm currently working on a series of optimizations that will help a bit, getting us closer to the theoretical speed limit. You might try installing gopls at master (git clone https://go.googlesource.com/tools && cd tools/gopls && go install), as it should be a bit better.

I'm also curious if your vim lsp client has partial results support. Is that something you could investigate?

[pand5a]

Thank you for your reply. I am on a long vacation. When I come back, I will try the code on the master.

I'm also curious if your vim lsp client has partial results support. Is that something you could investigate?

ok i will try it.

[pand5a]

From a code perspective, both nvim and VSCode currently do not support the 'Partial Result Progress' function for "completion". The following is the "PartialResultParams" parameter that is not specified by both nvim and VSCode when sending "completion" requests.

nvim:
https://github.com/neovim/neovim/blob/d5f6f61879bac3ac90512efe05d68e3500125a08/runtime/lua/vim/lsp/completion.lua#L385-L386
https://github.com/neovim/neovim/blob/master/runtime/lua/vim/lsp/util.lua#L1940-L1948

vscode:
https://github.com/microsoft/vscode-languageserver-node/blob/62b3198a95ac9af13e04a27248b28951064546f9/client/src/common/completion.ts#L121-L129
https://github.com/microsoft/vscode-languageserver-node/blob/62b3198a95ac9af13e04a27248b28951064546f9/client/src/common/codeConverter.ts#L343-L352

[pand5a]

@findleyr
One thing I don't quite understand is why typecheck is required for every completion. Is it to check if there are any errors?

[findleyr]

@pand5a sorry for the slow response. Type checking is required to resolve the lexical scope of the current package.

We could theoretically go faster if we only type checked the current function body+declarations, but we don't support that (it would require quite a hack around the current type checker APIs).

[pand5a]

Okay, I understand. Thank you very much.

[jheroy]

We could theoretically go faster if we only type checked the current function body+declarations, but we don't support that (it would require quite a hack around the current type checker APIs).

@findleyr Could you provide an idea or approach for a hack? I'd like to implement it myself for temporary personal use, Thanks.

[jheroy]

Thanks. There is currently a hard speed-limit for this completion, based on the fact that we cannot start streaming partial results (at least for VS Code: microsoft/vscode#105870), and therefore must await type checking of this very large package.

I'm also curious if your vim lsp client has partial results support. Is that something you could investigate?

Saghen/blink.cmp#395 (comment)
It seems that partial result is not implemented in gopls.