Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls: using 1GB RAM with Mattermost server #38903

Closed
ygworldr opened this issue May 6, 2020 · 19 comments
Closed

x/tools/gopls: using 1GB RAM with Mattermost server #38903

ygworldr opened this issue May 6, 2020 · 19 comments
Assignees
Milestone

Comments

@ygworldr
Copy link

@ygworldr ygworldr commented May 6, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.9 linux/amd64
$ gopls version
golang.org/x/tools/gopls 0.4.0
    golang.org/x/tools/gopls@v0.4.0 h1:G4+YP9kaV4dJb79J5MobyApxX493Qa6VoiTceUmxqik=

Does this issue reproduce with the latest release?

Unknown.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/yann/.cache/go-build"
GOENV="/home/yann/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/yann/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/lib/golang"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/yann/worldr/github/server/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build075671369=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Running gopls as part of vim-go. This is is one go file open.

What did you expect to see?

A small increase in memory.

What did you see instead?

; procs gopls
 PID:▲  User │ State Nice TTY CPU  MEM   VmPeak   VmSize   VmRSS TCP UDP  Read Write │ CPU Time Start             │ Command
             │                [%]  [%]  [bytes]  [bytes] [bytes]         [B/s] [B/s] │                            │
 212834 yann │ S        0     0.0 27.5   5.203G   5.203G  4.285G []  []  0.000 0.000 │ 00:01:58 2020/05/06 15:51  │ /home/yann/go/bin/gopls serve -listen unix;/tmp/gopls-e4935e-daemon.yann -listen.timeout 1m0s -logfile auto
 215897 yann │ S        0     0.0  0.1 767.355M 767.355M 18.465M []  []  0.000 0.000 │ 00:00:00 2020/05/06 15:53  │ /home/yann/go/bin/gopls -remote=auto
 216038 yann │ S        0     0.0  0.1 831.230M 767.293M 18.000M []  []  0.000 0.000 │ 00:00:00 2020/05/06 15:53  │ gopls -remote=auto

I have to constantly close/open neovim since otherwise, my machine dies because gopls takes all the 16G of RAM I have.

@gopherbot
Copy link

@gopherbot gopherbot commented May 6, 2020

Thank you for filing a gopls issue! Please take a look at the Troubleshooting guide, and make sure that you have provided all of the relevant information here.

@gopherbot gopherbot added the gopls label May 6, 2020
@stamblerre
Copy link
Contributor

@stamblerre stamblerre commented May 6, 2020

If you are using gopls/v0.4.0, you should see heap profiles in your temporary directory. If you find them, please upload them.

Edit: Sorry, I see you are using v0.4.0. Is this on a public or private repository? If it's public, do you mind sharing a link to it?

@stamblerre stamblerre changed the title gopls takes way too much memory and crashes the host machine. x/tools/gopls: using 16GB RAM May 6, 2020
@gopherbot gopherbot added this to the Unreleased milestone May 6, 2020
@gopherbot gopherbot added the Tools label May 6, 2020
@ygworldr
Copy link
Author

@ygworldr ygworldr commented May 7, 2020

The repository is the Mattermost server.

Here is a heap profile file (I hope!): gopls.149275-1GiB-heap.pb.gz

; procs gopls
 PID:▲  User │ State Nice TTY CPU MEM   VmPeak   VmSize   VmRSS TCP UDP  Read Write │ CPU Time Start             │ Command
             │                [%] [%]  [bytes]  [bytes] [bytes]         [B/s] [B/s] │                            │
 149275 yann │ S        0     0.0 8.4   2.269G   2.269G  1.312G []  []  0.000 0.000 │ 00:00:12 2020/05/07 12:18  │ /home/yann/go/bin/gopls serve -listen unix;/tmp/gopls-cbe194-daemon.yann -listen.timeout 1m0s -logfile auto
 150361 yann │ S        0     0.0 0.1 839.297M 839.297M 16.805M []  []  0.000 0.000 │ 00:00:00 2020/05/07 12:18  │ /home/yann/go/bin/gopls -remote=auto
 150503 yann │ S        0     0.0 0.1 695.227M 695.227M 15.723M []  []  0.000 0.000 │ 00:00:00 2020/05/07 12:18  │ gopls -remote=auto

@stamblerre
Copy link
Contributor

@stamblerre stamblerre commented May 8, 2020

Thank you for sharing this. It looks as we'd expect, with most of the memory allocated during type checking. By the way, it looks like this heap is only 1GiB, not 16 as you saw. What are your gopls settings - do you have staticcheck enabled by any chance?

Thank you for sharing the link to the repository - we'll try to reproduce. There are number of repos that seem to cause problems with gopls (#38752 is a similar issue, for example).

@agnivade agnivade changed the title x/tools/gopls: using 16GB RAM x/tools/gopls: using 1GB RAM May 8, 2020
@agnivade agnivade changed the title x/tools/gopls: using 1GB RAM x/tools/gopls: using 1GB RAM with Mattermost server May 8, 2020
@ygworldr
Copy link
Author

@ygworldr ygworldr commented May 8, 2020

@stamblerre Thank you so much for looking into it! Much appreciated.

Yes, it's not the one that generated it. I fiddled a little with dependencies and now I only have the version that vim-go specifically request: 0.4.0, without any additional tags. It seems to behave better but still gets to 4/5G. I will upload another large file when I get one.

do you have staticcheck enabled by any chance?

I do not think so, at least not explicitly. It might be turned on by default in vim-go.

@stamblerre
Copy link
Contributor

@stamblerre stamblerre commented May 8, 2020

Yes, it's not the one that generated it. I fiddled a little with dependencies and now I only have the version that vim-go specifically request: 0.4.0, without any additional tags. It seems to behave better but still gets to 4/5G. I will upload another large file when I get one.

Thank you!

I do not think so, at least not explicitly. It might be turned on by default in vim-go.

I don't believe vim-go turns on staticcheck by default, and based on the profile you attached that seems to be the case.

We'll follow up here when someone from the gopls team has a chance to take a closer look.

@ygworldr
Copy link
Author

@ygworldr ygworldr commented May 14, 2020

gopls.176223-3GiB-heap.pb.gz

Here's another heap, under gopls 0.4.1

@stamblerre stamblerre modified the milestones: Unreleased, gopls/v0.5.0 May 26, 2020
@heschik
Copy link
Contributor

@heschik heschik commented Jun 11, 2020

@ygworldr I believe some changes I landed today have improved memory usage on large projects substantially. Could you try master using the instructions at https://github.com/golang/tools/blob/master/gopls/doc/user.md#unstable-versions?

@ygworldr
Copy link
Author

@ygworldr ygworldr commented Jun 12, 2020

@heschik I updated the version I have and opened two files in the Mattermost project: channel_test.go and channel.go in api4. This is what I got:

√ ; gopls version
golang.org/x/tools/gopls master
    golang.org/x/tools/gopls@v0.0.0-20200612022331-742c5eb664c2 h1:+BNEvY32Dn2PtaE6qifJQQshGNIuJvcXAvYUDe6+/gc=
√ ; procs gopls
 PID:▲  User │ State Nice TTY   CPU MEM  VmPeak  VmSize    VmRSS TCP UDP  Read Write │ CPU Time Start             │ Command
             │                  [%] [%] [bytes] [bytes]  [bytes]         [B/s] [B/s] │                            │
 167625 yann │ S        0       0.0 0.1  1.242G  1.181G  16.816M []  []  0.000 0.000 │ 00:00:00 2020/06/12 13:17  │ /home/yann/go/bin/gopls -remote=auto
 167651 yann │ S        0     265.5 6.1  2.168G  2.168G 969.145M []  []  0.000 0.000 │ 00:00:08 2020/06/12 13:17  │ /home/yann/go/bin/gopls serve -listen unix;/tmp/gopls-c6d4b6-daemon.yann -listen.timeout 1m0s -logfile auto

I am afraid there was no heap file in /tmp… Do I need to do something special to generate it?

@heschik
Copy link
Contributor

@heschik heschik commented Jun 12, 2020

That looks like pretty good success, no? Your most recent report was 1.3G, and now it's down to 970M. Not a bad improvement.

The heaps are only generated once gopls crosses 1G so you won't have any yet.

I'm inclined to close this -- I don't see any sign that there's a specific problem with mattermost, and we've made significant progress since it was filed. Do you agree?

@heschik
Copy link
Contributor

@heschik heschik commented Jun 12, 2020

...assuming it doesn't grow up to 4 or 5 G again.

@ygworldr
Copy link
Author

@ygworldr ygworldr commented Jun 15, 2020

@heschik

I'm inclined to close this -- I don't see any sign that there's a specific problem with mattermost, and we've made significant progress since it was filed. Do you agree?

Yes, I think the improvements are great! Thank you so much for that. Your hard work is much appreciated!

@heschik heschik closed this Jun 15, 2020
@ygworldr
Copy link
Author

@ygworldr ygworldr commented Jun 18, 2020

@heschik, I am afraid I have some bad news… But they are not more than 3G so, maybe not so bad?

@heschik
Copy link
Contributor

@heschik heschik commented Jun 18, 2020

That's disappointing. Some progress but we should be able to do better.

I added a new diagnostics section to the debug server that may be helpful. If you follow the troubleshooting guide to start it, /cache/1 will have a table of per-package memory usage. Could you attach it here from a gopls that's using a lot of memory?

@heschik heschik reopened this Jun 18, 2020
@ygworldr
Copy link
Author

@ygworldr ygworldr commented Jun 19, 2020

That's disappointing. Some progress but we should be able to do better.

No worries whatsoever. I am really grateful for the work you are doing.

I added a new diagnostics section to the debug server that may be helpful. If you follow the troubleshooting guide to start it, /cache/1 will have a table of per-package memory usage. Could you attach it here from a gopls that's using a lot of memory?

I shall do as soon as is feasible.

@ygworldr
Copy link
Author

@ygworldr ygworldr commented Jun 19, 2020

Sadly, the log file is empty. The server command looks like:
/home/yann/go/bin/gopls serve -listen unix;/tmp/gopls-c6d4b6-daemon.yann -listen.timeout 1m0s -logfile auto and /home/yann/go/bin/gopls -remote=auto -rpc.trace

What am I doing wrong here?

@stamblerre stamblerre modified the milestones: gopls/v0.5.0, gopls/v1.0.0 Jul 22, 2020
@stamblerre
Copy link
Contributor

@stamblerre stamblerre commented Aug 12, 2020

There have been a number of memory improvements in gopls at master. If you can, please try out with the instructions on https://github.com/golang/tools/blob/master/gopls/doc/user.md#unstable-versions and let me know if that's any better.

@ygworldr
Copy link
Author

@ygworldr ygworldr commented Aug 12, 2020

@stamblerre I have updated just been doing lots of DevOps stuff and not much coding. This will change soon and I shall update. Thank you for your hard work!

@heschik heschik self-assigned this Sep 2, 2020
@stamblerre
Copy link
Contributor

@stamblerre stamblerre commented Sep 17, 2020

Closing, as this has been in the WaitingForInfo state for over a month. gopls/v0.5.0 has been released, and it contains these memory improvements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.