Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls: high memory consumption when used with a monorepo #37670

Closed
trapgate opened this issue Mar 4, 2020 · 43 comments
Closed

x/tools/gopls: high memory consumption when used with a monorepo #37670

trapgate opened this issue Mar 4, 2020 · 43 comments

Comments

@trapgate
Copy link

@trapgate trapgate commented Mar 4, 2020

Please answer these questions before submitting your issue. Thanks!

What did you do?

Our code is in a fairly large monorepo containing about 26K files, including vendored packages.
I use VSCode, with gopls enabled, and launched from the root of our monorepo, so the workspace includes the whole repo.

What did you expect to see?

gopls using 1-2GB of memory. That's a wild guess that I can't really validate, but 8-10 seems excessive.

What did you see instead?

The gopls process is using 8-10GB of memory. This number goes up and down with editor usage, but it never goes below about 6GB, and it generally hovers around 8GB. This is a 16GB laptop, so there's quite a bit of memory pressure when gopls is running.

Nearly all the code in our monorepo is intended for linux, and this is a mac, so I've experimented with adding this setting:

    "gopls": {
        "env": {"GOOS": "linux"}
    },

This does affect the number of errors reported by gopls, but the memory consumption is approximately unchanged.

I also tried building the latest master of gopls using the patch mentioned in this bug: #37223. This didn't help either. I'm attaching an SVG heap profile from that version.
pprof.zip

Build info

golang.org/x/tools/gopls v0.3.3
    golang.org/x/tools/gopls@v0.3.3 h1:mTFqRDJQmpSsgDDWvbtGnSva1z9uX2XcDszSWa6DhBQ=
    github.com/BurntSushi/toml@v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/sergi/go-diff@v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
    golang.org/x/mod@v0.1.1-0.20191105210325-c90efee705ee h1:WG0RUwxtNT4qqaXX3DPA8zHFNm/D9xaBpxzHt1WcA/E=
    golang.org/x/sync@v0.0.0-20190423024810-112230192c58 h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
    golang.org/x/tools@v0.0.0-20200227200655-6862ededa516 h1:OX66ZzpltgCOuBSGdaeT77hS2z3ub2AB+EuGxvGRBLE=
    golang.org/x/xerrors@v0.0.0-20191011141410-1b5146add898 h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
    honnef.co/go/tools@v0.0.1-2020.1.3 h1:sXmLre5bzIR6ypkjXCDI3jHPssRhc8KD/Ome589sc3U=
    mvdan.cc/xurls/v2@v2.1.0 h1:KaMb5GLhlcSX+e+qhbRJODnUUBvlw01jt4yrjFIHAuA=

Go info

go version go1.14 darwin/amd64

GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/geoffhickey/Library/Caches/go-build"
GOENV="/Users/geoffhickey/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY="*.internal.digitalocean.com,github.com/digitalocean"
GONOSUMDB="*.internal.digitalocean.com,github.com/digitalocean"
GOOS="darwin"
GOPATH="/Users/geoffhickey/do/cthulhu/docode"
GOPRIVATE="*.internal.digitalocean.com,github.com/digitalocean"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v7/w5lfw5qs1bd9np31z9vv04h80000gn/T/go-build021923438=/tmp/go-build -gno-record-gcc-switches -fno-common"
@gopherbot gopherbot added this to the Unreleased milestone Mar 4, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Mar 4, 2020

Thank you for filing a gopls issue! Please take a look at the Troubleshooting guide, and make sure that you have provided all of the relevant information here.

@heschik
Copy link
Contributor

@heschik heschik commented Mar 4, 2020

There should be some files named gopls.PID- in your temporary directory, named after heap sizes. Please upload the pair for the largest heap.

@trapgate
Copy link
Author

@trapgate trapgate commented Mar 4, 2020

@stamblerre stamblerre modified the milestones: Unreleased, gopls/v1.0.0 Mar 5, 2020
@heschik
Copy link
Contributor

@heschik heschik commented Mar 5, 2020

Thanks.

Unfortunately I don't have good news for you. Everything you've posted points to expected behavior -- I don't see any signs of a memory leak or work pileup. It looks like the monorepo is simply too large to work with all at once.

If you're only interested in one folder, you can start VS Code in that folder. If you're interested in multiple, you can add them piecemeal to your workspace, but note that most gopls features, e.g. find references, will only work within one folder.

@Jacalz
Copy link

@Jacalz Jacalz commented Apr 16, 2020

I am running in to the same issue described here and it is really annoying because my Linux machine with 8GB memory always runs out of memory and I have to pull out the power plug to be able to actually restart the computer.

This often occurs when i am using vscode and gopls with the https://github.com/fyne-io/fyne repository. I usually see memory slowly but steadily rising up to around five or six gigabytes until the memory in my computer is gone and everything stops functioning as expected. Killing of gopls just makes it rise steadily again.
gopls

@stamblerre
Copy link
Contributor

@stamblerre stamblerre commented Apr 16, 2020

@Jacalz: That sounds like a bug with older versions of gopls - can you confirm that you are using v0.4.0 (gopls version)?

It's also possible that you have multiple versions of gopls installed and VS Code is using an old one - you can confirm the path to the gopls binary that VS Code is using the Go: Locate Configured Go Tools command. Once we confirm you are using the correct gopls version, we can continue investigating the memory usage.

@Jacalz
Copy link

@Jacalz Jacalz commented Apr 16, 2020

@stamblerre: I verified that vscode is using the correct installation and this is the path of it: gopls: /home/jacob/go/bin/gopls installed. Looks good to me. The version should be v0.4.0 as long as vscode doesn't update to the wrong release. I tried running Go: Install/Update tools and I still have enormous amounts of memory usage.

@stamblerre
Copy link
Contributor

@stamblerre stamblerre commented Apr 16, 2020

You can confirm the version by running /home/jacob/go/bin/gopls version.

If the memory usage is high with gopls/v0.4.0, you should have some files named gopls.PID- in your temporary directory, named after heap sizes. Please upload the pair for the largest heap.

@Jacalz
Copy link

@Jacalz Jacalz commented Apr 17, 2020

Thanks. It indeed does appear like I am using the latest version:

jacob@pacman ~/go/src/sparta $ ~/go/bin/gopls version
golang.org/x/tools/gopls 0.4.0
    golang.org/x/tools/gopls@v0.4.0 h1:G4+YP9kaV4dJb79J5MobyApxX493Qa6VoiTceUmxqik=

I tried to find the PID files, but I could not find any. I looked in /tmp, but nothing with the name gopls stood out at all. Am I doing something wrong?

@stamblerre
Copy link
Contributor

@stamblerre stamblerre commented Apr 17, 2020

What OS are you on? Take a look at the https://pkg.go.dev/os?tab=doc#TempDir docs to see what directory gopls will write temporary files to.

@Jacalz
Copy link

@Jacalz Jacalz commented Apr 17, 2020

I am on Linux. I looks like I just wasn't getting into regions of memory usage that was concidered too much. I did not open the huge monorepo (https://github.com/fyne-io/fyne) and only saw 800MB memory usage due to that. Below are the files that you requested:

gopls.3169-5GiB-goroutines.txt
gopls.3169-5GiB-heap.pb.gz

I must admit that they were quite hard to get. When I tried for the first time, the memory usage jumped to 4GB and then gopls hugged 82% of my cpu and the computer totally froze and that meant that a force restart pruged the temporary files. This was by opening two files from https://github.com/fyne-io/fyne and doing small edits and a two saving of those files.

I restarted, had the system monitor open and ready to kill gopls when I had the chance. My system only has 8GB memory so when gopls starts using over 6GB (I saw a total of 6.7 GB before it was starting to get laggy and I killed it), things get really out of control and in to unplug power adapter territory. Thus I could only get the PID files for 5GB heaps at most. The PID files for 1, 2, 3, 4 GB heaps are saved too in case you need those too.

@heschik
Copy link
Contributor

@heschik heschik commented Apr 20, 2020

I was able to reproduce a similar profile by simply opening the root of the Fyne project. Most of the memory is being used by the type checker:

Showing nodes accounting for 853.95MB, 88.04% of 969.98MB total
      flat  flat%   sum%        cum   cum%
  620.02MB 63.92% 63.92%   620.02MB 63.92%  go/types.(*Checker).recordTypeAndValue
   87.50MB  9.02% 72.94%    98.44MB 10.15%  go/parser.(*parser).parseOperand
   43.13MB  4.45% 77.39%   153.17MB 15.79%  go/parser.(*parser).parseElementList

That's a little surprising for Fyne which is a relatively small project. My best guess is that the problem is the GL libraries, which generate tens of thousands of lines of cgo.

Regardless, I don't see a gopls bug here; it appears to be general go/types memory usage as usual.

@huapox
Copy link

@huapox huapox commented May 10, 2020

+1 with version gopls 0.4.0

@huapox
Copy link

@huapox huapox commented May 16, 2020

get worse with gopls 0.4.1

sam @ debian in /_ext/gopath/bin |03:24:50  
$ ./gopls version
golang.org/x/tools/gopls 0.4.1
    golang.org/x/tools/gopls@v0.4.1 h1:0e3BPxGV4B3cd0zdMuccwW72SgmHp92lAjOyxX/ScAw=
sam @ debian in /_ext/gopath/bin |03:32:22  
$ uname -a
Linux debian 4.9.0-4-amd64 #1 SMP Debian 4.9.51-1 (2017-09-28) x86_64 GNU/Linux

sam @ debian in /_ext/gopath/bin |03:43:33  
$ go version
go version go1.13.7 linux/amd64
@stamblerre stamblerre modified the milestones: gopls/v1.0.0, gopls/v0.5.0 May 21, 2020
@ggilley
Copy link

@ggilley ggilley commented May 31, 2020

I have 6 projects open in vscode. A gopls process for each is running ranging from 465MB to 900MB. Crazy!

@hunterlxt
Copy link

@hunterlxt hunterlxt commented Jun 1, 2020

gopls eat 22G memory within just one project. Do you have any experience with large project

image

using VSCode with the default config

@egorenar
Copy link

@egorenar egorenar commented Jun 1, 2020

syzkaller is another Go project which causes gopls on my machine to consume 27GiB RAM.

@Luffy110
Copy link

@Luffy110 Luffy110 commented Jun 3, 2020

i have the same issue .
my gopls version is
golang.org/x/tools/gopls 0.4.1
golang.org/x/tools/gopls@v0.4.1 h1:0e3BPxGV4B3cd0zdMuccwW72SgmHp92lAjOyxX/ScAw=

uname -a

Linux shenjie-VirtualBox 5.3.0-53-generic #47~18.04.1-Ubuntu SMP Thu May 7 13:10:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

go version :
go version go1.13.4 linux/amd64

@tommady
Copy link

@tommady tommady commented Jun 8, 2020

image

golang.org/x/tools/gopls 0.4.1
go version go1.14.2 darwin/amd64
34G too crazy 🤪

@agnivade
Copy link
Contributor

@agnivade agnivade commented Jun 8, 2020

Hi everyone, just a quick reminder that just mentioning your Go version and gopls version is not sufficient information to help us debug this further.

If you notice issues with gopls consuming high memory,

  • please include your VS Code settings
  • the repo that is causing this (if it's open source)
  • and a heap profile to go with it.

Thanks.

@Jacalz
Copy link

@Jacalz Jacalz commented Jun 21, 2020

Hi @heschik. Sorry for the late answer, I haven't had the time to look at it until now. I gave your changes a little spin with https://github.com/fyne-io/fyne and the difference is staggering. I will need to do more testing but initial results seem to indicate that memory usage caps out on around 4.2 GB instead of growing indefinitely until my computer gets starved of memory and crashes.

In short, it is finally usable even though memory usage still is absurdly high in my opinion. Will do some more testing before committing fully though :)

@Jacalz
Copy link

@Jacalz Jacalz commented Jun 22, 2020

Alright, it still seems to like to steal all my memory, but it seems to take more time before that happens. A good improvement definitely, but I still seem to need to upgrade the amount of ram in my computer to 16 GB Ram...

@trapgate
Copy link
Author

@trapgate trapgate commented Jun 23, 2020

Hi @heschik, the output from the debug server's /cache/1 endpoint would include a complete map of the directories in our repo, which I can't comfortably reveal. Here are some of the vendored packages from the top of the list though. This is from a gopls process that's currently using 9.5 GiB. I can probably share more of the information from the cache if you let me know what you're looking for.

vendor/github.com/swaggo/files (1)	35.68 = 35.63 + 0.02 + 0.01 + 0.02
vendor/github.com/digitalocean/go-netbox/netbox/client/dcim (1)	10.83 = 1.46 + 4.83 + 1.44 + 3.10
vendor/k8s.io/api/core/v1 (1)	10.12 = 2.24 + 3.92 + 0.97 + 3.01
@heschik
Copy link
Contributor

@heschik heschik commented Jun 23, 2020

@trapgate At high heaps, very recent gopls will write zip files to /tmp. The ones with -noname have the package names removed.

@trapgate
Copy link
Author

@trapgate trapgate commented Jun 24, 2020

Gotcha, here's one:

gopls.449451-7GiB-nonames.zip

@RayfenWindspear
Copy link

@RayfenWindspear RayfenWindspear commented Jul 2, 2020

Same thing here. Up to 9gb, but managed to trim it down to around 3.5gb. Pretty sure it was just less than valid paths and cleaning actual paths and cache files that brought it down.

Anyway, to the question at hand. What would be the viability of running a single instance of gopls against master of the repo and having everyone in the organization connect to it? There would be drift as work on your local branch progressed, but would it be able to handle, say, 50 people connecting to one instance remotely?

@heschik
Copy link
Contributor

@heschik heschik commented Jul 6, 2020

@RayfenWindspear Even if that were feasible I'm reasonably certain you'd just end up with a gopls instance using 50x the memory. There are bugs here, but massive server-side deployments are not going to be the fastest way to address them.

@RayfenWindspear
Copy link

@RayfenWindspear RayfenWindspear commented Jul 7, 2020

@heschik While I appreciate the reply, and forgive my ignorance if this is totally bogus, but would it not be able to reconcile and consolidate the memory for the indexed files? Or would it treat each individual Session as having its own separate set of indexed files? I do understand that it's technically pre-release and is listed as unstable, so perhaps that isn't part of its feature set (yet?).

Basically, we have several devs who already use VSCode to run remote to our dev server, which will each therefore run their own separate instance. In theory, even consolidating that to a single instance would at least allow it to not have to reel in the go.mod deps (disk space) and then index that all (memory) per instance.

So considering that, it seems like it could possibly be feasible and incur less overhead than one instance per dev. Sure it would still be O(n*repo), but (in theory, if it works like this), that's better than O(n* (repo+deps+instance_overhead) ).

Thoughts?

@waynr
Copy link

@waynr waynr commented Jul 20, 2020

So while tinkering around with go tool pprof and gopls on Friday evening, I was able to significantly reduce gopls memory usage in the same repo @trapgate is referring to in this issue by disabling the *debug.Instance when no -debug flag value is specified by the user. I estimate about a 15% reduction in memory consumption.

Another of my coworkers told me that emacs will prompt them for a working directory to pass to gopls which results in it consuming even less memory. Unfortunately I am a vim-go user and I'm not sure how to configure it to pass this kind of working directory hint to gopls. Is it something that can be configured by the editor plugin via Language Server Protocol?

I also experimented with moving the monorepo in question away from a monolithic module approach (currently our monorepo is comprised of several dozen projects all within the same go.mod) to a module-per-project approach and in combination with my abovementioned -debug "hack" saw my gopls memory consumption reduced by a total of roughly 70%.

@heschik
Copy link
Contributor

@heschik heschik commented Jul 20, 2020

alloc_space is total bytes allocated. Allocating memory isn't free, but if your goal is to reduce the overall heap size of gopls then you are looking at the wrong number.

Without looking carefully, I believe the primary result of your PR is to disable logging. My guess is that either you have verbose logging enabled, or something is going very very badly wrong in a way that results in log spam. Either way, please share the profile.

@waynr
Copy link

@waynr waynr commented Jul 20, 2020

@heschik thanks for the quick response! If I understand what you are saying correctly alloc_space isn't short for "currently allocated space" but instead for "total allocated during the process's lifetime" and at the point in time when the memory profile is captured that space may have been deallocated.

Without looking carefully, I believe the primary result of your PR is to disable logging.

The debug.Instance struct is what is used to keep track of package statistics; when a user passes for example -debug=localhost:6060 they can then view package statistics gathered by what i assumed to be hooks on the main language server processing bits that are passed to debug.WithInstance. I didn't realize that it also contributes to logging.

@waynr
Copy link

@waynr waynr commented Jul 20, 2020

I estimate about a 15% reduction in memory consumption.

After looking more carefully at the same profile in the PR I linked above, I see now what @heschik was talking about. Using go tool pprof -png -sample_index=inuse_space -nodefraction=0 -nodecount=500 gopls.profile.mem-with-debug-server.gz instead of go tool pprof -png -sample_index=alloc_space -nodefraction=0 -nodecount=500 gopls.profile.mem-with-debug-server.gz shows that the debug.Instance consumes an almost negligible amount of memory.

@stamblerre stamblerre modified the milestones: gopls/v0.5.0, gopls/v1.0.0 Jul 22, 2020
@stamblerre
Copy link
Contributor

@stamblerre stamblerre commented Aug 12, 2020

There have been a number of memory improvements in gopls at master. Please try out with the instructions on https://github.com/golang/tools/blob/master/gopls/doc/user.md#unstable-versions, or wait for the next release.

@Raghav-intrigue
Copy link

@Raghav-intrigue Raghav-intrigue commented Aug 19, 2020

Great improvement guys. My memory usage dropped from 9gigs to almost 2gb.

@trapgate
Copy link
Author

@trapgate trapgate commented Aug 21, 2020

For me at least, gopls 0.4.4 has been much more parsimonious than previous versions - I'm seeing 1-2GB of memory usage on my monorepo, where before I was seeing 8-10GB. I haven't yet tried running with master; I will try to check that out in the next couple of days and report back.

@stamblerre
Copy link
Contributor

@stamblerre stamblerre commented Aug 26, 2020

Glad to hear that things are working better! There will be more improvements coming in gopls/v0.5.0, so I think I will go ahead and close this issue for now. If you encounter memory usage issues again, please take a look at the instructions in #36943 and file a new issue.

@egorenar
Copy link

@egorenar egorenar commented Aug 28, 2020

I also confirm that it got much better, it became usable with large projects, thank you very much.

@rmerry
Copy link

@rmerry rmerry commented Oct 13, 2020

Using gopls at master hasn't fixed this issue for me.

I'm working on a monorepo (~40GB in size) and v0.4.4 would regularly cause my remote machine to become unresponsive by using all available memory. I'm now using master which has bought me a little more time but not fixed the issue.

$ gopls version
golang.org/x/tools/gopls master
    golang.org/x/tools/gopls@v0.0.0-20201013053347-2db1cd791039 h1:/32sBC1LOo43X5JHPUZT+hbLpyamXpL4FgR5eMEYb7w=

Gopls now takes quite a little while longer to eat up all my memory but if left unchecked it will do just that. I'm still having to kill it every 3-4 minutes. My solution now is to use a systemd watchdog to kill gopls when memory usage gets too high---not ideal but far better than losing connection to my remote box.

@Jacalz
Copy link

@Jacalz Jacalz commented Oct 13, 2020

I am having the roughly the same issues as @rmerry, but on a different repository. Using gopls with https://github.com/fyne-io/fyne results in massive memory consumption that easily climbs up to over 8 gigabytes of memory consumption. It climbs a lot more slowly with the latest releases, but it is still largely unusable sadly :(

@heschik
Copy link
Contributor

@heschik heschik commented Oct 13, 2020

@rmerry @Jacalz

Please file new issues following the steps at https://github.com/golang/tools/blob/master/gopls/doc/troubleshooting.md#memory-usage. Simply reporting that gopls uses a lot of memory, or describing the workarounds you use to kill it, doesn't help us solve the problem. The memory diagnostics might.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.