Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls: high memory consumption when used with a monorepo #37670

Open
trapgate opened this issue Mar 4, 2020 · 4 comments
Open

x/tools/gopls: high memory consumption when used with a monorepo #37670

trapgate opened this issue Mar 4, 2020 · 4 comments
Labels
Milestone

Comments

@trapgate
Copy link

@trapgate trapgate commented Mar 4, 2020

Please answer these questions before submitting your issue. Thanks!

What did you do?

Our code is in a fairly large monorepo containing about 26K files, including vendored packages.
I use VSCode, with gopls enabled, and launched from the root of our monorepo, so the workspace includes the whole repo.

What did you expect to see?

gopls using 1-2GB of memory. That's a wild guess that I can't really validate, but 8-10 seems excessive.

What did you see instead?

The gopls process is using 8-10GB of memory. This number goes up and down with editor usage, but it never goes below about 6GB, and it generally hovers around 8GB. This is a 16GB laptop, so there's quite a bit of memory pressure when gopls is running.

Nearly all the code in our monorepo is intended for linux, and this is a mac, so I've experimented with adding this setting:

    "gopls": {
        "env": {"GOOS": "linux"}
    },

This does affect the number of errors reported by gopls, but the memory consumption is approximately unchanged.

I also tried building the latest master of gopls using the patch mentioned in this bug: #37223. This didn't help either. I'm attaching an SVG heap profile from that version.
pprof.zip

Build info

golang.org/x/tools/gopls v0.3.3
    golang.org/x/tools/gopls@v0.3.3 h1:mTFqRDJQmpSsgDDWvbtGnSva1z9uX2XcDszSWa6DhBQ=
    github.com/BurntSushi/toml@v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/sergi/go-diff@v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
    golang.org/x/mod@v0.1.1-0.20191105210325-c90efee705ee h1:WG0RUwxtNT4qqaXX3DPA8zHFNm/D9xaBpxzHt1WcA/E=
    golang.org/x/sync@v0.0.0-20190423024810-112230192c58 h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
    golang.org/x/tools@v0.0.0-20200227200655-6862ededa516 h1:OX66ZzpltgCOuBSGdaeT77hS2z3ub2AB+EuGxvGRBLE=
    golang.org/x/xerrors@v0.0.0-20191011141410-1b5146add898 h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
    honnef.co/go/tools@v0.0.1-2020.1.3 h1:sXmLre5bzIR6ypkjXCDI3jHPssRhc8KD/Ome589sc3U=
    mvdan.cc/xurls/v2@v2.1.0 h1:KaMb5GLhlcSX+e+qhbRJODnUUBvlw01jt4yrjFIHAuA=

Go info

go version go1.14 darwin/amd64

GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/geoffhickey/Library/Caches/go-build"
GOENV="/Users/geoffhickey/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY="*.internal.digitalocean.com,github.com/digitalocean"
GONOSUMDB="*.internal.digitalocean.com,github.com/digitalocean"
GOOS="darwin"
GOPATH="/Users/geoffhickey/do/cthulhu/docode"
GOPRIVATE="*.internal.digitalocean.com,github.com/digitalocean"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v7/w5lfw5qs1bd9np31z9vv04h80000gn/T/go-build021923438=/tmp/go-build -gno-record-gcc-switches -fno-common"
@gopherbot gopherbot added this to the Unreleased milestone Mar 4, 2020
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Mar 4, 2020

Thank you for filing a gopls issue! Please take a look at the Troubleshooting guide, and make sure that you have provided all of the relevant information here.

@heschik

This comment has been minimized.

Copy link
Contributor

@heschik heschik commented Mar 4, 2020

There should be some files named gopls.PID- in your temporary directory, named after heap sizes. Please upload the pair for the largest heap.

@trapgate

This comment has been minimized.

Copy link
Author

@trapgate trapgate commented Mar 4, 2020

@stamblerre stamblerre modified the milestones: Unreleased, gopls/v1.0.0 Mar 5, 2020
@heschik

This comment has been minimized.

Copy link
Contributor

@heschik heschik commented Mar 5, 2020

Thanks.

Unfortunately I don't have good news for you. Everything you've posted points to expected behavior -- I don't see any signs of a memory leak or work pileup. It looks like the monorepo is simply too large to work with all at once.

If you're only interested in one folder, you can start VS Code in that folder. If you're interested in multiple, you can add them piecemeal to your workspace, but note that most gopls features, e.g. find references, will only work within one folder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.