Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: 1.12 compiler needs double the memory than 1.11 #32089

pmuetschard opened this issue May 16, 2019 · 3 comments

cmd/compile: 1.12 compiler needs double the memory than 1.11 #32089

pmuetschard opened this issue May 16, 2019 · 3 comments


Copy link

@pmuetschard pmuetschard commented May 16, 2019

What version of Go are you using (go version)?

1.12.4 vs 1.11.2


We ( generate go code based off our own domain specific language. We generate 2 go packages each with about 500k LOC. I have observed that the memory needed by the go compiler for these packages has doubled in 1.12 compared to 1.11. In 1.11, the compile executable used between 3.5G and 4G of RAM for each package, while the 1.12 version needs ~8G (resident memory, virtual memory is even more). I have observed this on both X86_64 Linux and MacOS. This causes our build bots to run out of memory and the compile fails with the below stack trace.

I don't know if you can fix this, but I felt that doubling the memory usage is a significant enough performance regression you might want to know about.

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xe5ddf9, 0x16)
	/usr/local/go/src/runtime/panic.go:617 +0x72
runtime.sysMap(0xc184000000, 0x4000000, 0x16c9178)
	/usr/local/go/src/runtime/mem_linux.go:170 +0xc7
runtime.(*mheap).sysAlloc(0x16a17c0, 0x4000, 0x16a17d0, 0x2)
	/usr/local/go/src/runtime/malloc.go:633 +0x1cd
runtime.(*mheap).grow(0x16a17c0, 0x2, 0x0)
	/usr/local/go/src/runtime/mheap.go:1232 +0x42
runtime.(*mheap).allocSpanLocked(0x16a17c0, 0x2, 0x16c9188, 0x7f5385d8bab0)
	/usr/local/go/src/runtime/mheap.go:1150 +0x3a7
runtime.(*mheap).alloc_m(0x16a17c0, 0x2, 0xc183fc0048, 0x7f5385d8bab0)
	/usr/local/go/src/runtime/mheap.go:977 +0xc2
	/usr/local/go/src/runtime/mheap.go:1048 +0x4c
	/usr/local/go/src/runtime/asm_amd64.s:351 +0x66

goroutine 1 [running]:
	/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc11639f4c8 sp=0xc11639f4c0 pc=0x458f70
runtime.(*mheap).alloc(0x16a17c0, 0x2, 0x7f5386010048, 0x7f5385d8bab0)
	/usr/local/go/src/runtime/mheap.go:1047 +0x8a fp=0xc11639f518 sp=0xc11639f4c8 pc=0x423faa
runtime.(*mcentral).grow(0x16a2d40, 0x0)
	/usr/local/go/src/runtime/mcentral.go:256 +0x95 fp=0xc11639f560 sp=0xc11639f518 pc=0x416ee5
runtime.(*mcentral).cacheSpan(0x16a2d40, 0x190)
	/usr/local/go/src/runtime/mcentral.go:106 +0x2ff fp=0xc11639f5c0 sp=0xc11639f560 pc=0x4169ef
runtime.(*mcache).refill(0x7f539a2cd008, 0x48)
	/usr/local/go/src/runtime/mcache.go:135 +0x86 fp=0xc11639f5e0 sp=0xc11639f5c0 pc=0x416486
runtime.(*mcache).nextFree(0x7f539a2cd008, 0xf6bf01e5f765ea48, 0x40869d, 0xc11639fa98, 0x1fa2ae7604efec75)
	/usr/local/go/src/runtime/malloc.go:786 +0x88 fp=0xc11639f618 sp=0xc11639f5e0 pc=0x40b3c8
runtime.mallocgc(0x700, 0xe0fd00, 0x1, 0x1)
	/usr/local/go/src/runtime/malloc.go:939 +0x76e fp=0xc11639f6b8 sp=0xc11639f618 pc=0x40bcde
runtime.newarray(0xe0fd00, 0x4, 0xc183ffc000)
	/usr/local/go/src/runtime/malloc.go:1085 +0x63 fp=0xc11639f6e8 sp=0xc11639f6b8 pc=0x40c1d3
runtime.makeBucketArray(0xddd400, 0xcdf62a782fd61c02, 0x0, 0xb71308, 0xc11639f920)
	/usr/local/go/src/runtime/map.go:364 +0x18e fp=0xc11639f720 sp=0xc11639f6e8 pc=0x40d0ce
runtime.hashGrow(0xddd400, 0xc11639f928)
	/usr/local/go/src/runtime/map.go:1035 +0x8b fp=0xc11639f770 sp=0xc11639f720 pc=0x40eddb
runtime.mapassign(0xddd400, 0xc11639f928, 0xc11639f900, 0x16c7160)
	/usr/local/go/src/runtime/map.go:654 +0x147 fp=0xc11639f7f8 sp=0xc11639f770 pc=0x40da37
	/usr/local/go/src/cmd/compile/internal/ssa/zcse.go:24 +0x281 fp=0xc11639faf8 sp=0xc11639f7f8 pc=0xb6ad31
	/usr/local/go/src/cmd/compile/internal/ssa/compile.go:90 +0x479 fp=0xc1163a3578 sp=0xc11639faf8 pc=0x637519
cmd/compile/internal/gc.buildssa(0xc0181f49a0, 0x0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:233 +0xbfd fp=0xc1163a36e8 sp=0xc1163a3578 pc=0xca6f0d
cmd/compile/internal/gc.compileSSA(0xc0181f49a0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:299 +0x4d fp=0xc1163a3798 sp=0xc1163a36e8 pc=0xc71add
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:278 +0x5b1 fp=0xc1163a3838 sp=0xc1163a3798 pc=0xc71a01
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:221 +0xc2 fp=0xc1163a3890 sp=0xc1163a3838 pc=0xc71342
	/usr/local/go/src/cmd/compile/internal/gc/main.go:665 +0x3059 fp=0xc1163a3f20 sp=0xc1163a3890 pc=0xc48f39
	/usr/local/go/src/cmd/compile/main.go:51 +0xad fp=0xc1163a3f98 sp=0xc1163a3f20 pc=0xd8401d
	/usr/local/go/src/runtime/proc.go:200 +0x20c fp=0xc1163a3fe0 sp=0xc1163a3f98 pc=0x42e58c
	/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc1163a3fe8 sp=0xc1163a3fe0 pc=0x45aec1

This comment has been minimized.

Copy link

@randall77 randall77 commented May 16, 2019

Do you have a self-contained test I can run? The full build of gapid looks huge and has lots of non-Go dependencies.
Have you tried tip?


This comment has been minimized.

Copy link

@bradfitz bradfitz commented May 16, 2019

@pmuetschard, the time when we could've done anything to fix Go 1.12 was about 5 months ago. That tree is pretty locked down at this point. But we're mid-freeze and still stabilizing Go 1.13 now (tip == the HEAD of our current master branch), so testing that would be useful.

But any fixes there would almost certainly be out of scope for our backporting policy. (That's reserved for critical & security bugs)

@bradfitz bradfitz changed the title 1.12 compiler needs double the memory than 1.11 cmd/compile: 1.12 compiler needs double the memory than 1.11 May 16, 2019
@josharian josharian added the ToolSpeed label May 16, 2019

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jun 16, 2019

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this Jun 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
5 participants
You can’t perform that action at this time.