Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls: cannot handle large files #33003

Open
myitcv opened this issue Jul 9, 2019 · 3 comments

Comments

@myitcv
Copy link
Member

commented Jul 9, 2019

What version of Go are you using (go version)?

$ go version
go version devel +13327f219e Sat Jul 6 13:25:59 2019 +0000 linux/amd64
$ go list -m golang.org/x/tools
golang.org/x/tools v0.0.0-20190708171232-dd7c7173f160
$ go list -m golang.org/x/tools/gopls
golang.org/x/tools/gopls v0.1.2-0.20190708171232-dd7c7173f160

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOENV="/home/myitcv/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/myitcv/gos"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/myitcv/gostuff/src/github.com/myitcv/govim/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build907756658=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Open $(go env GOROOT)/src/cmd/compile/internal/ssa/rewriteAMD64.go in your editor, and remove the indentation from every line in the file. Now trigger goimports by whatever means necessary in your editor; you will see gopls run out of memory before the method returns:

 fatal error: runtime: out of memory
 
 runtime stack:
 runtime.throw(0xadb7c1, 0x16)
 	/home/myitcv/gos/src/runtime/panic.go:758 +0x72
 runtime.sysMap(0xc210000000, 0x4000000, 0xfd5438)
 	/home/myitcv/gos/src/runtime/mem_linux.go:169 +0xc5
 runtime.(*mheap).sysAlloc(0xfbcfa0, 0x20e000, 0x18000376a8, 0x8ceaa0)
 	/home/myitcv/gos/src/runtime/malloc.go:681 +0x1cd
 runtime.(*mheap).grow(0xfbcfa0, 0x107, 0xffffffff)
 	/home/myitcv/gos/src/runtime/mheap.go:1244 +0x42
 runtime.(*mheap).allocSpanLocked(0xfbcfa0, 0x107, 0xfd5448, 0xf742c0)
 	/home/myitcv/gos/src/runtime/mheap.go:1155 +0x291
 runtime.(*mheap).alloc_m(0xfbcfa0, 0x107, 0x101, 0x0)
 	/home/myitcv/gos/src/runtime/mheap.go:1007 +0xc2
 runtime.(*mheap).alloc.func1()
 	/home/myitcv/gos/src/runtime/mheap.go:1078 +0x4c
 runtime.(*mheap).alloc(0xfbcfa0, 0x107, 0x7fc6c3010101, 0x4292d0)
 	/home/myitcv/gos/src/runtime/mheap.go:1077 +0x8a
 runtime.largeAlloc(0x20d6c8, 0x450101, 0xc20fdc4000)
 	/home/myitcv/gos/src/runtime/malloc.go:1103 +0x97
 runtime.mallocgc.func1()
 	/home/myitcv/gos/src/runtime/malloc.go:998 +0x46
 runtime.systemstack(0xc00029c180)
 	/home/myitcv/gos/src/runtime/asm_amd64.s:370 +0x66
 runtime.mstart()
 	/home/myitcv/gos/src/runtime/proc.go:1146
 
 goroutine 2805 [running]:
 runtime.systemstack_switch()
 	/home/myitcv/gos/src/runtime/asm_amd64.s:330 fp=0xc008b19308 sp=0xc008b19300 pc=0x45da60
 runtime.mallocgc(0x20d6c8, 0x9c8c00, 0xfffffffffffff601, 0xc20fdc4000)
 	/home/myitcv/gos/src/runtime/malloc.go:997 +0x85c fp=0xc008b193a8 sp=0xc008b19308 pc=0x40e19c
 runtime.makeslice(0x9c8c00, 0x41ad9, 0x41ad9, 0xc20fdc4000)
 	/home/myitcv/gos/src/runtime/slice.go:49 +0x6c fp=0xc008b193d8 sp=0xc008b193a8 pc=0x4469fc
 golang.org/x/tools/internal/lsp/diff.shortestEditSequence(0xc047034000, 0x106b6, 0x106b7, 0xc04713c000, 0x106b6, 0x106b7, 0x1, 0x1, 0x7fc6a6638a90, 0x106b6)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/diff/diff.go:177 +0x394 fp=0xc008b19460 sp=0xc008b193d8 pc=0x8cf5f4
 golang.org/x/tools/internal/lsp/diff.Operations(0xc047034000, 0x106b6, 0x106b7, 0xc04713c000, 0x106b6, 0x106b7, 0xc00002a2d0, 0x28, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/diff/diff.go:70 +0x81 fp=0xc008b19528 sp=0xc008b19460 pc=0x8ceaa1
 golang.org/x/tools/internal/lsp/source.computeTextEdits(0xbb9ca0, 0xc0250a66f0, 0xbbc4a0, 0xc0000ca3c0, 0xc046d96000, 0x15db55, 0x0, 0x0, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/source/format.go:145 +0x391 fp=0xc008b19628 sp=0xc008b19528 pc=0x8e66b1
 golang.org/x/tools/internal/lsp/source.Imports(0xbb9ca0, 0xc0250a66f0, 0xbc3380, 0xc000272340, 0xbc2080, 0xc0000ca3c0, 0xc0001abe00, 0xd4d18, 0x2134f4, 0x0, ...)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/source/format.go:83 +0x518 fp=0xc008b19748 sp=0xc008b19628 pc=0x8e5b58
 golang.org/x/tools/internal/lsp.organizeImports(0xbb9ca0, 0xc0250a66f0, 0xbc3380, 0xc000272340, 0xc0000301e0, 0x44, 0x1, 0x1, 0x0, 0x1, ...)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/code_action.go:103 +0x131 fp=0xc008b19800 sp=0xc008b19748 pc=0x92df41
 golang.org/x/tools/internal/lsp.(*Server).codeAction(0xc000224000, 0xbb9ca0, 0xc0250a66f0, 0xc0244752c0, 0xc0244752c0, 0x0, 0x0, 0x0, 0xc0249aae70)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/code_action.go:48 +0x3ad fp=0xc008b19ba8 sp=0xc008b19800 pc=0x92d27d
 golang.org/x/tools/internal/lsp.(*Server).CodeAction(0xc000224000, 0xbb9ca0, 0xc0250a66f0, 0xc0244752c0, 0xc0244752c0, 0x0, 0x0, 0x493ff2, 0xc0001af860)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/server.go:204 +0x4d fp=0xc008b19c00 sp=0xc008b19ba8 pc=0x938bdd
 golang.org/x/tools/internal/lsp/protocol.serverHandler.func1(0xbb9ca0, 0xc0250a66f0, 0xc02508a060)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/protocol/tsserver.go:355 +0x2b1a fp=0xc008b19f00 sp=0xc008b19c00 pc=0x73929a
 golang.org/x/tools/internal/jsonrpc2.(*Conn).Run.func1(0xc0001d2000, 0xc02508a060, 0xc000226000, 0xbb9ca0, 0xc0250a66f0, 0xc0250956d0, 0xc006df3d00)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/jsonrpc2/jsonrpc2.go:441 +0x15c fp=0xc008b19fa8 sp=0xc008b19f00 pc=0x72cfdc
 runtime.goexit()
 	/home/myitcv/gos/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc008b19fb0 sp=0xc008b19fa8 pc=0x45fb31
 created by golang.org/x/tools/internal/jsonrpc2.(*Conn).Run
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/jsonrpc2/jsonrpc2.go:428 +0x746
 
 goroutine 1 [syscall]:
 syscall.Syscall(0x0, 0x0, 0xc000222000, 0x1000, 0xc000239438, 0x800000, 0x42f74a)
 	/home/myitcv/gos/src/syscall/asm_linux_amd64.s:18 +0x5
 syscall.read(0x0, 0xc000222000, 0x1000, 0x1000, 0x0, 0x0, 0x6)
 	/home/myitcv/gos/src/syscall/zsyscall_linux_amd64.go:732 +0x5a
 syscall.Read(...)
 	/home/myitcv/gos/src/syscall/syscall_unix.go:187
 internal/poll.(*FD).Read(0xc0000b0000, 0xc000222000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
 	/home/myitcv/gos/src/internal/poll/fd_unix.go:165 +0x164
 os.(*File).read(...)
 	/home/myitcv/gos/src/os/file_unix.go:259
 os.(*File).Read(0xc0000ae000, 0xc000222000, 0x1000, 0x1000, 0xc0250a09b5, 0xae7715, 0x1)
 	/home/myitcv/gos/src/os/file.go:112 +0x71
 bufio.(*Reader).fill(0xc0001af8c0)
 	/home/myitcv/gos/src/bufio/bufio.go:100 +0x103
 bufio.(*Reader).ReadSlice(0xc0001af8c0, 0xa, 0x8dfc45d58af49759, 0xc000239628, 0x40a95b, 0xc0250a3070, 0x8dfc45d58af49759)
 	/home/myitcv/gos/src/bufio/bufio.go:359 +0x3d
 bufio.(*Reader).ReadBytes(0xc0001af8c0, 0xa, 0xc000239600, 0xc0001cc7e0, 0xc0001cc8a8, 0xaff1a8, 0x7fc6ddad2b28)
 	/home/myitcv/gos/src/bufio/bufio.go:437 +0x73
 bufio.(*Reader).ReadString(...)
 	/home/myitcv/gos/src/bufio/bufio.go:477
 golang.org/x/tools/internal/jsonrpc2.(*headerStream).Read(0xc0001a89a0, 0xbb9c20, 0xc0000aa000, 0xc02508a060, 0xc000226000, 0xbb9ca0, 0xc0250a66f0, 0xc0250956d0, 0xc006df3d00)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/jsonrpc2/stream.go:97 +0x92
 golang.org/x/tools/internal/jsonrpc2.(*Conn).Run(0xc000226000, 0xbb9c20, 0xc0000aa000, 0xc0001a89e0, 0xc000224000)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/jsonrpc2/jsonrpc2.go:396 +0xa2
 golang.org/x/tools/internal/lsp.(*Server).Run(...)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/server.go:61
 golang.org/x/tools/internal/lsp/cmd.(*Serve).Run(0xc00021a030, 0xbb9c20, 0xc0000aa000, 0xc0000b2030, 0x0, 0x0, 0x0, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/cmd/serve.go:95 +0x515
 golang.org/x/tools/internal/tool.Main.func2(0x0, 0xbbc520, 0xc00021a030, 0xbb9c20, 0xc0000aa000, 0xc0001af800, 0x0, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/tool/tool.go:130 +0xda
 golang.org/x/tools/internal/tool.Main(0xbb9c20, 0xc0000aa000, 0xbbc520, 0xc00021a030, 0xc0000b2030, 0x0, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/tool/tool.go:131 +0x241
 golang.org/x/tools/internal/lsp/cmd.(*Application).Run(0xc00021a000, 0xbb9c20, 0xc0000aa000, 0xc0000b2030, 0x0, 0x0, 0x7, 0x199)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/cmd/cmd.go:106 +0x48c
 golang.org/x/tools/internal/tool.Main.func2(0xc00021a000, 0xbbc4e0, 0xc00021a000, 0xbb9c20, 0xc0000aa000, 0xc0001af7a0, 0x0, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/tool/tool.go:130 +0xda
 golang.org/x/tools/internal/tool.Main(0xbb9c20, 0xc0000aa000, 0xbbc4e0, 0xc00021a000, 0xc0000b2010, 0x3, 0x3)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/tool/tool.go:131 +0x241
 main.main()
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools/gopls@v0.1.2-0.20190706070813-72ffa07ba3db/main.go:20 +0xba

This was discovered whilst I was trying to test out how govim behaved under the "load" of thousands of edits from gopls.

What did you expect to see?

The file to be correctly reformatted.

What did you see instead?

Out of memory error.


cc @stamblerre @ianthehat

@gopherbot gopherbot added this to the Unreleased milestone Jul 9, 2019

@stamblerre stamblerre changed the title x/tools/cmd/gopls: out of memory error on large file with lots of required formats x/tools/gopls: cannot handle large files Aug 8, 2019

@stamblerre

This comment has been minimized.

Copy link
Contributor

commented Aug 8, 2019

Seems like gopls in general just cannot handle a file of this size. Running gopls -rpc.trace -v check rewriteAMD64.go times out.

@ianthehat

This comment has been minimized.

Copy link

commented Aug 19, 2019

We seem to be able to handle large files mostly now, the editor works, we can survive them being type checked and generating outlines (although it is slow and we seem to redo it too much even when the file is not changing) but the original test case still fails.
Undenting the whole file causes almost every line to be different, which causes diff.shortestEditSequence to allocate huge amounts of memory, and eventually the machine runs out (it takes a while).
Given that we were looking to change diff algorithms anyway to something that supported sub-line, I am not sure how much effort we should put into fixing this rather than just finding a good diff library and switching to it.
Does anyone know of a good one?

@gopherbot

This comment has been minimized.

Copy link

commented Aug 21, 2019

Change https://golang.org/cl/191018 mentions this issue: gopls: use go-diff for edit generation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.