Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls: high memory consumption #54268

Open
codisms opened this issue Aug 4, 2022 · 4 comments
Open

x/tools/gopls: high memory consumption #54268

codisms opened this issue Aug 4, 2022 · 4 comments
Labels
gopls Tools
Milestone

Comments

@codisms
Copy link

codisms commented Aug 4, 2022

gopls version

Build info
----------
golang.org/x/tools/gopls v0.9.1
    golang.org/x/tools/gopls@v0.9.1 h1:SigsTL4Hpv3a6b/a7oPCLRv5QUeSM6PZNdta1oKY4B0=
    github.com/google/go-cmp@v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/mod@v0.6.0-dev.0.20220419223038-86c51ed26bb4 h1:6zppjxzCulZykYSLyVDYbneBfbaBIQPYMevg0bEwv2s=
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
    golang.org/x/sys@v0.0.0-20220209214540-3681064d5158 h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c=
    golang.org/x/text@v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
    golang.org/x/tools@v0.1.12-0.20220713141851-7464a5a40219 h1:Ljlba2fVWOA1049JjsKii44g8nZN2GjpxMlzVc8AnQM=
    mvdan.cc/gofumpt@v0.3.0 h1:kTojdZo9AcEYbQYhGuLf/zszYthRdhDNDUi2JKTxas4=
    mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: go1.16.15

go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOENV="/home/ubuntu/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ubuntu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ubuntu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.15"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3575468840=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Editing Go files with vim and vim-go.

What did you expect to see?

Lower memory consumption.

What did you see instead?

Over 5GB of memory consumption; thrashing swap file.

Editor and settings

VIM - Vi IMproved 9.0 (2022 Jun 28, compiled Aug  1 2022 05:17:14)
Included patches: 1-124
Compiled by ubuntu@dev-veritone
Huge version without GUI.  Features included (+) or not (-):
+acl               +file_in_path      +mouse_urxvt       -tag_any_white
+arabic            +find_in_path      +mouse_xterm       -tcl
+autocmd           +float             +multi_byte        +termguicolors
+autochdir         +folding           +multi_lang        +terminal
-autoservername    -footer            -mzscheme          +terminfo
-balloon_eval      +fork()            +netbeans_intg     +termresponse
+balloon_eval_term +gettext           +num64             +textobjects
-browse            -hangul_input      +packages          +textprop
++builtin_terms    +iconv             +path_extra        +timers
+byte_offset       +insert_expand     +perl/dyn          +title
+channel           +ipv6              +persistent_undo   -toolbar
+cindent           +job               +popupwin          +user_commands
-clientserver      +jumplist          +postscript        +vartabs
-clipboard         +keymap            +printer           +vertsplit
+cmdline_compl     +lambda            +profile           +vim9script
+cmdline_hist      +langmap           -python            +viminfo
+cmdline_info      +libcall           +python3/dyn       +virtualedit
+comments          +linebreak         +quickfix          +visual
+conceal           +lispindent        +reltime           +visualextra
+cryptv            +listcmds          +rightleft         +vreplace
+cscope            +localmap          +ruby/dyn          +wildignore
+cursorbind        -lua               +scrollbind        +wildmenu
+cursorshape       +menu              +signs             +windows
+dialog_con        +mksession         +smartindent       +writebackup
+diff              +modify_fname      -sodium            -X11
+digraphs          +mouse             -sound             -xfontset
-dnd               -mouseshape        +spell             -xim
-ebcdic            +mouse_dec         +startuptime       -xpm
+emacs_tags        -mouse_gpm         +statusline        -xsmp
+eval              -mouse_jsbterm     -sun_workshop      -xterm_clipboard
+ex_extra          +mouse_netterm     +syntax            -xterm_save
+extra_search      +mouse_sgr         +tag_binary
-farsi             -mouse_sysmouse    -tag_old_static
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -g -O2 -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc -L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-E -L/usr/local/lib -Wl,--as-needed -o vim -lm -ltinfo -lrt -ldl -Wl,-E -fstack-protector-strong -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu/perl/5.30/CORE -lperl -ldl -lm -lpthread -lcrypt

Logs

gopls.3782930-5GiB-nonames.zip

@gopherbot gopherbot added Tools gopls labels Aug 4, 2022
@gopherbot gopherbot added this to the Unreleased milestone Aug 4, 2022
@findleyr
Copy link
Contributor

findleyr commented Aug 5, 2022

That gopls uses a lot of memory is a known (and difficult) problem. See for example #47855.

However, 5gb is a lot, and we should check whether that makes sense based on your usage. Can you tell me:

  • Approximately how many .go files are in your workspace (this depends on which folder was resolved as your workspace folder).
  • Do you experience this memory usage immediately, or does it take a while to build up? There was a large memory leak that will be fixed in the next gopls release (v0.9.2), so I'm curious if you may be experiencing that.

Unfortunately, at the moment 5gb is not unheard-of memory usage when working on large projects, such as Kubernetes. We're working on this, but it is a long-term project.

@findleyr findleyr added the WaitingForInfo label Aug 5, 2022
@codisms
Copy link
Author

codisms commented Aug 5, 2022

Unfortunately, I don't have a great way to answer the first question. The number is hard to know because of the libraries being referenced, but the project itself has 3500 Go files, though not all referenced in all parts of the code. One thing I can say is that we have a single file generated by gqlgen that has 256k lines in it and is probably a big culprit.

Initially loading the project causes the machine to spin for a good five minutes or so and it'll spike to 3-4 GB pretty easily. It'll settle around 2-3 after a while but spikes as the files are edited. Multiple files could be changed at the same time with multiple vim windows going on. I do have the gopls server running for quite a long time though.

@findleyr
Copy link
Contributor

findleyr commented Aug 5, 2022

Thanks.

I expect you'll see it stay in the the ~3gb range after gopls@v0.9.2 is released. I'll take a look at the heap dump you've provided to see if I can understand this better.

@findleyr findleyr modified the milestones: Unreleased, gopls/later Aug 8, 2022
@findleyr findleyr removed the WaitingForInfo label Aug 8, 2022
@findleyr findleyr modified the milestones: gopls/later, gopls/v0.10.0 Aug 8, 2022
@findleyr
Copy link
Contributor

findleyr commented Aug 8, 2022

Putting this in the next milestone for additional triage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
gopls Tools
Projects
None yet
Development

No branches or pull requests

3 participants