x/tools/gopls: poor performance with very large packages in import graph

[sff2578]

What version of Go, VS Code & VS Code Go extension are you using?

Version Information

• Run go version to get version of Go from the VS Code integrated terminal.

  • go version go1.18.10 linux/amd64

• Run gopls -v version to get version of Gopls from the VS Code integrated terminal.

  • golang.org/x/tools/gopls v0.12.3
    golang.org/x/tools/gopls@v0.12.3 h1:u0wCI9uvt7mnmri6bFBIaWw1XCN6PN8hKv55Zwd+GbE=

• Run code -v or code-insiders -v to get version of VS Code or VS Code Insiders.

• Check your installed extensions to get the version of the VS Code Go extension

  • v0.39.0

• Run Ctrl+Shift+P (Cmd+Shift+P on Mac OS) > Go: Locate Configured Go Tools command.

  • go: /usr/local/go1.18.10/bin/go: go version go1.18.10 linux/amd64

  gotests: /usr/local/gopkgs/bin/gotests (version: v1.6.0 built with go: go1.18.10)
  gomodifytags: /usr/local/gopkgs/bin/gomodifytags (version: v1.16.0 built with go: go1.18.10)
  impl: /usr/local/gopkgs/bin/impl (version: v1.1.0 built with go: go1.18.10)
  goplay: /usr/local/gopkgs/bin/goplay (version: v1.0.0 built with go: go1.18.10)
  dlv: /usr/local/gopkgs/bin/dlv (version: v1.20.2 built with go: go1.18.10)
  staticcheck: /usr/local/gopkgs/bin/staticcheck (version: v0.3.3 built with go: go1.18.10)
  gopls: /usr/local/gopkgs/bin/gopls (version: v0.12.3 built with go: go1.18.10)

Share the Go related settings you have added/edited

Run Preferences: Open Settings (JSON) command to open your settings.json file.
Share all the settings with the go. or ["go"] or gopls prefixes.

Describe the bug

i manually installed gopls with version v0.11.0, but after some time(less than an hour), go extension will automatically update it to the latest version. I'm having issue with latest gopls(comsuming too much CPU and slow) so i need to stay with older version

Steps to reproduce the behavior:

1. manually install gopls v0.11.0
2. connect vscode to server, work for a while
3. observer gopls is upgraded, in the "go output" of vscode, saw logs:

Installing 1 tool at /usr/local/gopkgs/bin in module mode.
  gopls@0.12.3

Installing golang.org/x/tools/gopls@v0.12.3 (/usr/local/gopkgs/bin/gopls) SUCCEEDED

Screenshots or recordings

If applicable, add screenshots or recordings to help explain your problem.

[findleyr]

I'm having issue with latest gopls(comsuming too much CPU and slow) so i need to stay with older version

Can you please explain more? The latest gopls should not be appreciably slower than v0.11.0. If you are observing this, it would really help us if you could share more of what you see. Are you working in an open-source project? What do you perceive as being slow? When do you notice the CPU?

[sff2578]

I'm not working on an open source project, i rely on gopls to do auto formatting and auto complete, with version v0.12.3, when i save, it take about 10-15 seconds to complete the save. Up on save, a notice pop up saying: Saving xxx.go:Getting code actions from "Go"
also when i use an interface, waiting for the suggestion dropdown list, it takes even longer. Meantime i have a top running to see the usage, gopls sometimes take 500% of CPU or even go up to 1000%.
with v0.11.0, this won't happen, everything happens within a secode and CPU is not that high

[findleyr]

@sff2578 what you describe is a bug, plain and simple. We want gopls@v0.12+ to perform approximately the same as v0.11.0, using much less memory (and in the future, we want it to perform better than v0.11.0 across the board).

I know of certain areas where we could reduce CPU in v0.12.x (#60926), but those would be say a 50% reduction in CPU while typing. I wasn't aware of any order-of-magnitude regressions. We would greatly appreciate it if you would work with us to help us understand the problems you are facing. This could help many other users.

To prevent auto-upgrade, you can use the following setting, but as mentioned above we hope that you will work with us to resolve your problems with v0.12.x:
https://github.com/golang/vscode-go/wiki/settings#gotoolsmanagementautoupdate

Can you please run the gopls stats -anon command using gopls@latest, and share your results? That will give us a sense of the size of your repo. The -anon flag should remove any output that could contain file paths, but please do inspect the json blob to make sure it is something you are comfortable sharing.

[findleyr]

CC @adonovan, who can also help track down this regression.

[sff2578]

Thanks for the pointer, it now works fine without upgrading gopls, here's the output of the gopls stats -anon command:

{
  "DirStats": {
    "Files": 706,
    "TestdataFiles": 0,
    "GoFiles": 606,
    "ModFiles": 1,
    "Dirs": 23
  },
  "GOARCH": "amd64",
  "GOOS": "linux",
  "GoVersion": "go1.18.10",
  "GoplsVersion": "v0.12.4",
  "InitialWorkspaceLoadDuration": "10.497067387s",
  "MemStats": {
    "HeapAlloc": 1640464000,
    "HeapInUse": 1990836224,
    "TotalAlloc": 4751304536
  },
  "WorkspaceStats": {
    "Files": {
      "Total": 7815,
      "Largest": 4950165,
      "Errs": 0
    },
    "Views": [
      {
        "GoCommandVersion": "go1.18.10",
        "AllPackages": {
          "Packages": 1073,
          "LargestPackage": 514,
          "CompiledGoFiles": 7880,
          "Modules": 176
        },
        "WorkspacePackages": {
          "Packages": 9,
          "LargestPackage": 505,
          "CompiledGoFiles": 673,
          "Modules": 1
        },
        "Diagnostics": 56
      }
    ]
  }
}

[findleyr]

Thanks @sff2578, that's really useful.

I notice something about your workspace, which may contribute tot he problem. According to that stats command, you have a package with 505 files in it. There are certain aspects of our new implementation that may perform very poorly on such huge packages.

Are you often working in that package? Did you notice the performance problems more significantly when working in that package?

[sff2578]

Thanks for the reply. i do heavily rely on this package, i use the interfaces and functions in this package a lot. i tried manually created a simple package, it is not hitting this issue

[adonovan]

I just ran an ecosystem metrics query for large packages, and quickly found several with >500 files. For example, this one has 672: https://github.com/oracle/oci-go-sdk/blob/master/dataintegration/. I ran gopls v0.12 on it and observed that it was indeed slow, especially to save a file, though still quite usable. A CPU profile showed time spent in parsing, type-checking, and analysis, but not typerefs. (It also showed the usual exaggerated hotspot of filecache.gc because of macOS setitimer syscall skew. I need to try it again on Linux)

[findleyr]

I'm going to repurpose this issue to track fixing performance with large packages.