Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls: optimize didChange handling #45686

Open
findleyr opened this issue Apr 22, 2021 · 8 comments
Open

x/tools/gopls: optimize didChange handling #45686

findleyr opened this issue Apr 22, 2021 · 8 comments

Comments

@findleyr
Copy link
Contributor

@findleyr findleyr commented Apr 22, 2021

Recent profiling has shown that, particularly on large codebases, snapshot cloning can significantly impact gopls' responsiveness.

Snapshot cloning is not optimized: there are lots of opportunities for performance improvement. For example:

  • URI.Filename() is a huge source of unnecessary cost; almost all URI manipulations could be done directly with the URI string rather than converting to file paths (which involves URI parsing).
  • Most of the maps in the snapshot could be replaced with an alternative data structure, optimized for fast cloning. For example, with an overlay that is occasionally compactified.
  • The import graph doesn't need to be rebuilt from scratch.
  • Known directories could be memoized.
  • The explicit inheritance of cache keys could probably instead be achieved with an asynchronous sweep.

Not sure which of these we'll do. Filing this as an umbrella issue to track improving performance.

CC @heschi @stamblerre

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 22, 2021

Change https://golang.org/cl/312689 mentions this issue: internal/lsp/cache: preallocate internal maps when cloning snapshots

@justplesh
Copy link
Contributor

@justplesh justplesh commented Apr 22, 2021

Working on unnecessary URI.Filename() usage in snapshot reduction

justplesh added a commit to justplesh/tools that referenced this issue Apr 22, 2021
The existing implementation uses a lot of URI.Filename() calls,
which are pretty expensive. Moreover, these calls are not necessary,
as long as all the actions could be done with the raw URI string.
This patch removes such calls and uses simple string casts.

Updates golang/go#45686
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 22, 2021

Change https://golang.org/cl/312809 mentions this issue: internal/lsp/cache: improve snapshot clone perfomance

justplesh added a commit to justplesh/tools that referenced this issue Apr 22, 2021
The existing implementation uses a lot of URI.Filename() calls,
which are pretty expensive. Moreover, these calls are not necessary,
as long as all the actions could be done with the raw URI string.
This patch removes such calls and uses simple string casts.

Updates golang/go#45686
@findleyr findleyr changed the title x/tools/gopls: optimize snapshot cloning x/tools/gopls: optimize didChange handling Apr 23, 2021
@findleyr
Copy link
Contributor Author

@findleyr findleyr commented Apr 23, 2021

Extending this more generally to didChange handling, as snapshot.clone is only one of the problems. There is some other stuff in e.g. didModifyFiles that's quite slow.

@justplesh thanks very much for your contribution, and interest. Right now we're still trying to figure out the best path forward for some of these optimizations. Hopefully we'll have a better sense of what needs to be prioritized next week. If you'd like, we can keep you in mind for anything that is relatively self-contained.

@justplesh
Copy link
Contributor

@justplesh justplesh commented Apr 23, 2021

@findleyr Sure, please mention me or assign any lsp issue that I may work on. I'll be happy to work on it on some +- regular basis.

@justplesh
Copy link
Contributor

@justplesh justplesh commented Apr 23, 2021

@findleyr will we merge my PR then or will we wait for some more fundamental approach?

gopherbot pushed a commit to golang/tools that referenced this issue Apr 23, 2021
For large codebases, the cost of copying these maps can be fairly high,
especially when it needs to repeatedly grow the map's underlying storage.
Preallocate these to the size of the original snapshot maps to prevent
the need to grow the storage during the clone.

Updates golang/go#45686

Change-Id: I4cfcd5b7cba8110e4f7e706fd9ea968aaeb6ff0c
Reviewed-on: https://go-review.googlesource.com/c/tools/+/312689
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
@findleyr
Copy link
Contributor Author

@findleyr findleyr commented Apr 25, 2021

@justplesh we can proceed with your CL; it should be an improvement.

gopherbot pushed a commit to golang/tools that referenced this issue Apr 26, 2021
The existing implementation uses a lot of URI.Filename() calls,
which are pretty expensive. Moreover, these calls are not necessary,
as long as all the actions could be done with the raw URI string.
This patch removes such calls and uses simple string casts.

Updates golang/go#45686

Change-Id: Ibe11735969eaf0cfe33024f08418e14bf71e7fc4
GitHub-Last-Rev: 67a3ccd
GitHub-Pull-Request: #306
Reviewed-on: https://go-review.googlesource.com/c/tools/+/312809
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
Trust: Rebecca Stambler <rstambler@golang.org>
Trust: Suzy Mueller <suzmue@golang.org>
Run-TryBot: Rebecca Stambler <rstambler@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented May 5, 2021

Change https://golang.org/cl/317292 mentions this issue: internal/lsp/regtest: add a benchmark for didChange

gopherbot pushed a commit to golang/tools that referenced this issue May 6, 2021
Add a benchmark for the processing of workspace/didChange notifications,
attempting to isolate the synchronous change processing from
asynchronous diagnostics. To enable this, add a new type of expectation
that asserts on work that has been _started_, but not necessarily
completed. Of course, what we really want to know is whether the current
notification has been processed, but that's ~equivalent to knowing
whether the next one has been started. Really, it's off-by-one, but
amortized over e.g. the 100 iterations of a benchmark we get
approximately the right results.

Also change some functions to accept testing.TB, because in a first pass
at this I modified the regtest framework to operate on testing.B in
addition to testing.T... but that didn't work out as IWL is just too
slow to execute the benchmarks outside of the environment -- even though
we can ResetTimer, the benchmark execution is just too slow to be
usable. It seems like a fine change to accept testing.TB is some places,
though.

For golang/go#45686

Change-Id: I8894444b01177dc947bbed56ec7df80a15a2eae9
Reviewed-on: https://go-review.googlesource.com/c/tools/+/317292
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants