Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: out of memory with large slice and map composite literals #22010

Closed
h12w opened this issue Sep 25, 2017 · 7 comments

Comments

Projects
None yet
6 participants
@h12w
Copy link

commented Sep 25, 2017

What version of Go are you using (go version)?

go version go1.9 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/tmp.B05GgQUqzW/go"
GORACE=""
GOROOT="/usr/local/go1.9"
GOTOOLDIR="/usr/local/go1.9/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build029171386=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

go get h12.me/ua
go build -v h12.me/ua

What did you expect to see?

go build finishes as go1.8.3.

What did you see instead?

h12.me/ua
# h12.me/ua
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xb62665, 0x16)
	/usr/local/go/src/runtime/panic.go:605 +0x95
runtime.sysMap(0xc45a290000, 0x2f0000, 0xc420060100, 0xf05378)
	/usr/local/go/src/runtime/mem_linux.go:216 +0x1d0
runtime.(*mheap).sysAlloc(0xee3f40, 0x2f0000, 0xc42006fe20)
	/usr/local/go/src/runtime/malloc.go:470 +0xd7
runtime.(*mheap).grow(0xee3f40, 0x172, 0x0)
	/usr/local/go/src/runtime/mheap.go:887 +0x60
runtime.(*mheap).allocSpanLocked(0xee3f40, 0x172, 0xf05388, 0x7fe3c6a9fa18)
	/usr/local/go/src/runtime/mheap.go:800 +0x334
runtime.(*mheap).alloc_m(0xee3f40, 0x172, 0xffffffffffff0100, 0xc420060180)
	/usr/local/go/src/runtime/mheap.go:666 +0x118
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:733 +0x4d
runtime.systemstack(0xc42006ff08)
	/usr/local/go/src/runtime/asm_amd64.s:360 +0xab
runtime.(*mheap).alloc(0xee3f40, 0x172, 0x7fe3c6010100, 0x7fe3c6a9fa18)
	/usr/local/go/src/runtime/mheap.go:732 +0xa1
runtime.largeAlloc(0x2e2ba0, 0x7fe3c8220001, 0x7fe3c6a9fa18)
	/usr/local/go/src/runtime/malloc.go:827 +0x98
runtime.mallocgc.func1()
	/usr/local/go/src/runtime/malloc.go:722 +0x46
runtime.systemstack(0xc420020600)
	/usr/local/go/src/runtime/asm_amd64.s:344 +0x79
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1125

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:298 fp=0xc43ea636e8 sp=0xc43ea636e0 pc=0x455170
runtime.mallocgc(0x2e2ba0, 0xb2bf20, 0x17101, 0xc43c652228)
	/usr/local/go/src/runtime/malloc.go:721 +0x7b8 fp=0xc43ea63790 sp=0xc43ea636e8 pc=0x410638
runtime.makeslice(0xb2bf20, 0x1715d, 0x1715d, 0x1715d, 0x0, 0x80)
	/usr/local/go/src/runtime/slice.go:54 +0x77 fp=0xc43ea637c0 sp=0xc43ea63790 pc=0x43f157
cmd/compile/internal/ssa.makeLCArange(0xc43c652140, 0x0)
	/usr/local/go/src/cmd/compile/internal/ssa/lca.go:36 +0x8d fp=0xc43ea63920 sp=0xc43ea637c0 pc=0x5e783d
cmd/compile/internal/ssa.tighten(0xc43c652140)
	/usr/local/go/src/cmd/compile/internal/ssa/tighten.go:52 +0x2a6 fp=0xc43ea63a98 sp=0xc43ea63920 pc=0x8ff876
cmd/compile/internal/ssa.Compile(0xc43c652140)
	/usr/local/go/src/cmd/compile/internal/ssa/compile.go:70 +0x2bb fp=0xc43ea673d0 sp=0xc43ea63a98 pc=0x5cb19b
cmd/compile/internal/gc.buildssa(0xc43cae8b00, 0x0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:216 +0xd89 fp=0xc43ea67530 sp=0xc43ea673d0 pc=0x9f6b29
cmd/compile/internal/gc.compileSSA(0xc43cae8b00, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:240 +0x3c fp=0xc43ea675a8 sp=0xc43ea67530 pc=0x9c363c
cmd/compile/internal/gc.compile(0xc43cae8b00)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:219 +0x218 fp=0xc43ea67608 sp=0xc43ea675a8 pc=0x9c3568
cmd/compile/internal/gc.funccompile(0xc43cae8b00)
	/usr/local/go/src/cmd/compile/internal/gc/dcl.go:1049 +0xb7 fp=0xc43ea67660 sp=0xc43ea67608 pc=0x96a007
cmd/compile/internal/gc.fninit(0xc4203abe00, 0x28, 0x40)
	/usr/local/go/src/cmd/compile/internal/gc/init.go:207 +0xb59 fp=0xc43ea67880 sp=0xc43ea67660 pc=0x98fe39
cmd/compile/internal/gc.Main(0xb73f90)
	/usr/local/go/src/cmd/compile/internal/gc/main.go:592 +0x2c78 fp=0xc43ea67f08 sp=0xc43ea67880 pc=0x99e578
main.main()
	/usr/local/go/src/cmd/compile/main.go:49 +0x95 fp=0xc43ea67f80 sp=0xc43ea67f08 pc=0xabdfb5
runtime.main()
	/usr/local/go/src/runtime/proc.go:185 +0x20d fp=0xc43ea67fe0 sp=0xc43ea67f80 pc=0x42bedd
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc43ea67fe8 sp=0xc43ea67fe0 pc=0x457cf1
@mvdan

This comment has been minimized.

Copy link
Member

commented Sep 25, 2017

Both 1.9 and tip indeed seem to run out of memory. I have 7GB of free memory and it takes the build around 10s to use it all up, so it looks like it's stuck in an endless loop somewhere.

Will try to investigate and get a smaller reproduction program.

@mvdan

This comment has been minimized.

Copy link
Member

commented Sep 25, 2017

Nope - removing the 23k elements from the handsets map fixes it on my machine. Removing all elements of deviceInfoMap also fixes it. But leaving both eats up my memory.

I think I saw issues about huge composite literals (especially really long slices and maps) using too much CPU time or memory. I wonder what could have caused the regression from 1.8. A bit beyond me, I'm afraid.

CC @mdempsky @randall77

@mvdan mvdan added this to the Go1.10 milestone Sep 25, 2017

@mvdan mvdan changed the title go build out of memory cmd/compile: out of memory with large slice and map composite literals Sep 25, 2017

@gopherbot

This comment has been minimized.

Copy link

commented Sep 25, 2017

Change https://golang.org/cl/66050 mentions this issue: cmd/compile: improve static map initialization

@randall77

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2017

CL 66050 reduces memory use from ~15GB to ~0.5GB.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2017

That is only 96% smaller. Why stop there?

@mdempsky

This comment has been minimized.

Copy link
Member

commented Sep 25, 2017

@ianlancetaylor Gotta leave some work for 1.11.

@gopherbot gopherbot closed this in 91cb9ed Sep 25, 2017

@randall77

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2017

This is now fixed on tip.
What I don't understand is why 1.8 is better than 1.9.
1.8 uses mapassign and 1.9 uses mapassign_faststr, but I don't see how that would make much of a difference. The generated assembly is otherwise very similar.

Since this is a regression from 1.8 to 1.9, there's a possibility we could backport this fix (or another fix more directly related to the 1.8->1.9 regression, if we can understand it) to 1.9.1. But I'm leaning against doing this, it is a difficult bug to trigger (probably requires generated code) and can be worked around (doing the optimization in CL66050 manually).

@golang golang locked and limited conversation to collaborators Sep 25, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.