Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fpTracebackPCs crash on linux/amd64 in various places #59692

Closed
mknyszek opened this issue Apr 18, 2023 · 24 comments
Closed

runtime: fpTracebackPCs crash on linux/amd64 in various places #59692

mknyszek opened this issue Apr 18, 2023 · 24 comments
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@mknyszek
Copy link
Contributor

mknyszek commented Apr 18, 2023

I was running the Sweet benchmarks and I encountered a crash in fpTracebackPCs:

Crash in Go compiler during the go-build benchmark building kubelet.
SIGSEGV: segmentation violation
PC=0x458f97 m=2 sigcode=128

goroutine 0 [idle]:
runtime.fpTracebackPCs(...)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:944
runtime.traceStackID(0x1e32c9ce48d12?, {0x7f957d0a7018, 0x7f9590097100?, 0x80}, 0xffffffffffff8000?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:916 +0x217 fp=0xc000075cf8 sp=0xc000075cb0 pc=0x458f97
runtime.traceEventLocked(0x42606b?, 0x459960?, 0x0?, 0xc0000485d0, 0x12, 0x0, 0x1, {0x0, 0x0, 0x0})
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:759 +0x285 fp=0xc000075d70 sp=0xc000075cf8 pc=0x458565
runtime.traceEvent(0x0?, 0x1, {0x0, 0x0, 0x0})
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:691 +0xa9 fp=0xc000075dd8 sp=0xc000075d70 pc=0x458269
runtime.traceGoPreempt(...)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:1534
runtime.gopreempt_m(0x1416360?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:3550 +0x45 fp=0xc000075e18 sp=0xc000075dd8 pc=0x43fb05
runtime.newstack()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/stack.go:1072 +0x3cb fp=0xc000075fc8 sp=0xc000075e18 pc=0x44f92b
runtime.morestack()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:593 +0x8f fp=0xc000075fd0 sp=0xc000075fc8 pc=0x46872f

goroutine 864 [running]:
runtime.traceEvent(0x2c?, 0xffffffffffffffff, {0x0, 0x0, 0x0})
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:668 +0xe8 fp=0xc0044c71a0 sp=0xc0044c7198 pc=0x4582a8
runtime.traceGCMarkAssistDone(...)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:1496
runtime.gcAssistAlloc(0xc002444340)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgcmark.go:516 +0x27d fp=0xc0044c7200 sp=0xc0044c71a0 pc=0x41dc9d
runtime.deductAssistCredit(0x4?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/malloc.go:1220 +0x54 fp=0xc0044c7228 sp=0xc0044c7200 pc=0x40c914
runtime.mallocgc(0x5c0, 0xcf5820, 0x1)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/malloc.go:935 +0xc9 fp=0xc0044c7290 sp=0xc0044c7228 pc=0x40c149
runtime.newarray(0xc0047a6306?, 0xc0047a6460?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/malloc.go:1279 +0x45 fp=0xc0044c72b8 sp=0xc0044c7290 pc=0x40ca65
runtime.makeBucketArray(0xcc1b40?, 0xd0?, 0xa4b0d6?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/map.go:364 +0x18d fp=0xc0044c72f8 sp=0xc0044c72b8 pc=0x40d80d
runtime.hashGrow(0xc008a9b428?, 0xc008afb4d0)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/map.go:1068 +0x72 fp=0xc0044c7338 sp=0xc0044c72f8 pc=0x40f292
runtime.mapassign(0xcc1b40, 0xc008afb4d0, 0xc0033d3200?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/map.go:659 +0xf3 fp=0xc0044c73c0 sp=0xc0044c7338 pc=0x40dff3
cmd/compile/internal/ssa.(*slotCanonicalizer).lookup(0xc008a9b578, {0xc008ae84d0, 0xc0001fa480, 0x0, 0xc008af1bc0, 0x0})
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/debug.go:396 +0x25b fp=0xc0044c7488 sp=0xc0044c73c0 pc=0x7059fb
cmd/compile/internal/ssa.PopulateABIInRegArgOps(0xc004493520)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/debug.go:446 +0x12f fp=0xc0044c76e8 sp=0xc0044c7488 pc=0x705baf
cmd/compile/internal/ssa.BuildFuncDebug(0xcc0520?, 0xc004493520, 0x1100000000?, 0x1b?, 0xc0019d5c20)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/debug.go:578 +0x1cb fp=0xc0044c7898 sp=0xc0044c76e8 pc=0x706c8b
cmd/compile/internal/ssagen.genssa(0xc004493520, 0xc008ac7260)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/ssa.go:7225 +0xf7d fp=0xc0044c7ea8 sp=0xc0044c7898 pc=0xa99d7d
cmd/compile/internal/ssagen.Compile(0xc0033d14a0, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/pgen.go:197 +0x26f fp=0xc0044c7f70 sp=0xc0044c7ea8 pc=0xa5b58f
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:184 +0x34 fp=0xc0044c7fb0 sp=0xc0044c7f70 pc=0xc46954
cmd/compile/internal/gc.compileFunctions.func3.1()
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:166 +0x30 fp=0xc0044c7fe0 sp=0xc0044c7fb0 pc=0xc46d70
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0044c7fe8 sp=0xc0044c7fe0 pc=0x46a441
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 24
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:165 +0x23a

goroutine 1 [semacquire]:
runtime.gopark(0x20?, 0xc0000389b8?, 0x0?, 0xb2?, 0x4053e0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc003e13738 sp=0xc003e13718 pc=0x43900e
runtime.goparkunlock(...)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:392
runtime.semacquire1(0xc003fa23f8, 0xf8?, 0x1, 0x0, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/sema.go:160 +0x20f fp=0xc003e137a8 sp=0xc003e13738 pc=0x449caf
sync.runtime_Semacquire(0xc00338be40?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/sema.go:62 +0x25 fp=0xc003e137e0 sp=0xc003e137a8 pc=0x466c45
sync.(*WaitGroup).Wait(0xc000b70800?)
	../../../../tip+trace+nanotime/bin/goroot/src/sync/waitgroup.go:116 +0x48 fp=0xc003e13808 sp=0xc003e137e0 pc=0x4832c8
cmd/compile/internal/gc.compileFunctions()
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:196 +0x21d fp=0xc003e13860 sp=0xc003e13808 pc=0xc467bd
cmd/compile/internal/gc.Main(0xd754f0)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/main.go:369 +0x1c6f fp=0xc003e13ee0 sp=0xc003e13860 pc=0xc4906f
main.main()
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/main.go:57 +0xf9 fp=0xc003e13f40 sp=0xc003e13ee0 pc=0xc6ff79
runtime.main()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:255 +0x2b2 fp=0xc003e13fe0 sp=0xc003e13f40 pc=0x438bb2
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc003e13fe8 sp=0xc003e13fe0 pc=0x46a441

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005efa8 sp=0xc00005ef88 pc=0x43900e
runtime.goparkunlock(...)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:392
runtime.forcegchelper()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:310 +0xb3 fp=0xc00005efe0 sp=0xc00005efa8 pc=0x438e73
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005efe8 sp=0xc00005efe0 pc=0x46a441
created by runtime.init.6 in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:298 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005f778 sp=0xc00005f758 pc=0x43900e
runtime.goparkunlock(...)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:392
runtime.bgsweep(0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgcsweep.go:319 +0xdf fp=0xc00005f7c8 sp=0xc00005f778 pc=0x423fbf
runtime.gcenable.func1()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:178 +0x25 fp=0xc00005f7e0 sp=0xc00005f7c8 pc=0x419745
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005f7e8 sp=0xc00005f7e0 pc=0x46a441
created by runtime.gcenable in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:178 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00007e000?, 0xe8bcd0?, 0x0?, 0x0?, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005ff70 sp=0xc00005ff50 pc=0x43900e
runtime.goparkunlock(...)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:392
runtime.(*scavengerState).park(0x13e90e0)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgcscavenge.go:399 +0x49 fp=0xc00005ffa0 sp=0xc00005ff70 pc=0x421f49
runtime.bgscavenge(0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgcscavenge.go:632 +0x59 fp=0xc00005ffc8 sp=0xc00005ffa0 pc=0x4224f9
runtime.gcenable.func2()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:179 +0x25 fp=0xc00005ffe0 sp=0xc00005ffc8 pc=0x4196e5
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x46a441
created by runtime.gcenable in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:179 +0xa5

goroutine 17 [finalizer wait]:
runtime.gopark(0x0?, 0xc00013a4f8?, 0xa0?, 0x21?, 0x1000000010?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00011ee28 sp=0xc00011ee08 pc=0x43900e
runtime.runfinq()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mfinal.go:193 +0x107 fp=0xc00011efe0 sp=0xc00011ee28 pc=0x4187e7
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00011efe8 sp=0xc00011efe0 pc=0x46a441
created by runtime.createfing in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mfinal.go:163 +0x3d

goroutine 18 [trace reader (blocked)]:
runtime.gopark(0xc00005a778?, 0xc00012fce0?, 0x18?, 0x24?, 0xfbb9?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005a738 sp=0xc00005a718 pc=0x43900e
runtime.ReadTrace()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:445 +0x31 fp=0xc00005a7a0 sp=0xc00005a738 pc=0x4579b1
runtime/trace.Start.func1()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace/trace.go:130 +0x45 fp=0xc00005a7e0 sp=0xc00005a7a0 pc=0xc45e65
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x46a441
created by runtime/trace.Start in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace/trace.go:128 +0xdf

goroutine 24 [select]:
runtime.gopark(0xc00005afb0?, 0x2?, 0x0?, 0x0?, 0xc00005af6c?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005adc8 sp=0xc00005ada8 pc=0x43900e
runtime.selectgo(0xc00005afb0, 0xc00005af68, 0x30?, 0x0, 0xca4f40?, 0x1)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/select.go:327 +0x725 fp=0xc00005aee0 sp=0xc00005adc8 pc=0x448d05
cmd/compile/internal/gc.compileFunctions.func3()
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:154 +0x126 fp=0xc00005afe0 sp=0xc00005aee0 pc=0xc46b26
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005afe8 sp=0xc00005afe0 pc=0x46a441
created by cmd/compile/internal/gc.compileFunctions in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:147 +0xe7

goroutine 5 [GC worker (idle)]:
runtime.gopark(0x142dea0?, 0x3?, 0x13?, 0x82?, 0xbfd8e0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005b750 sp=0xc00005b730 pc=0x43900e
runtime.gcBgMarkWorker()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc00005b7e0 sp=0xc00005b750 pc=0x41b385
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005b7e8 sp=0xc00005b7e0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c

goroutine 33 [GC worker (idle)]:
runtime.gopark(0x0?, 0xbfd880?, 0x40?, 0x45?, 0xbfd8e0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005bf50 sp=0xc00005bf30 pc=0x43900e
runtime.gcBgMarkWorker()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc00005bfe0 sp=0xc00005bf50 pc=0x41b385
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005bfe8 sp=0xc00005bfe0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c

goroutine 23 [GC worker (idle)]:
runtime.gopark(0x1e32c84698de9?, 0x3?, 0x60?, 0x59?, 0xbfd8e0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005c750 sp=0xc00005c730 pc=0x43900e
runtime.gcBgMarkWorker()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc00005c7e0 sp=0xc00005c750 pc=0x41b385
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c

goroutine 6 [GC worker (idle)]:
runtime.gopark(0x1e32c7ea7231e?, 0x3?, 0xb8?, 0xea?, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc000060750 sp=0xc000060730 pc=0x43900e
runtime.gcBgMarkWorker()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc0000607e0 sp=0xc000060750 pc=0x41b385
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x142dea0?, 0x1?, 0x3d?, 0x2c?, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc000060f50 sp=0xc000060f30 pc=0x43900e
runtime.gcBgMarkWorker()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc000060fe0 sp=0xc000060f50 pc=0x41b385
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000060fe8 sp=0xc000060fe0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x142dea0?, 0x3?, 0xa7?, 0x84?, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc000061750 sp=0xc000061730 pc=0x43900e
runtime.gcBgMarkWorker()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc0000617e0 sp=0xc000061750 pc=0x41b385
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0000617e8 sp=0xc0000617e0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x142dea0?, 0x3?, 0x41?, 0x91?, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc003db0750 sp=0xc003db0730 pc=0x43900e
runtime.gcBgMarkWorker()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc003db07e0 sp=0xc003db0750 pc=0x41b385
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc003db07e8 sp=0xc003db07e0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x142dea0?, 0x1?, 0x60?, 0xc6?, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc003db0f50 sp=0xc003db0f30 pc=0x43900e
runtime.gcBgMarkWorker()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc003db0fe0 sp=0xc003db0f50 pc=0x41b385
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc003db0fe8 sp=0xc003db0fe0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c

goroutine 890 [runnable]:
runtime.mapaccess2_fast64(0xcc2b00, 0xc008c27c08, 0x6e7)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/map_fast64.go:53 +0x185 fp=0xc008c27968 sp=0xc008c27960 pc=0x410b85
cmd/compile/internal/ssa.numberLines.func1(...)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/numberlines.go:114
cmd/compile/internal/ssa.numberLines(0xc00262bd40)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/numberlines.go:136 +0x209 fp=0xc008c27ec8 sp=0xc008c27968 pc=0x743229
cmd/compile/internal/ssa.Compile(0xc00262bd40)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/compile.go:97 +0x95e fp=0xc008c2bb80 sp=0xc008c27ec8 pc=0x6f81be
cmd/compile/internal/ssagen.buildssa(0xc00343f4a0, 0x1)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/ssa.go:567 +0x1fc7 fp=0xc008c2bea8 sp=0xc008c2bb80 pc=0xa63ee7
cmd/compile/internal/ssagen.Compile(0xc00343f4a0, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/pgen.go:187 +0x45 fp=0xc008c2bf70 sp=0xc008c2bea8 pc=0xa5b365
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:184 +0x34 fp=0xc008c2bfb0 sp=0xc008c2bf70 pc=0xc46954
cmd/compile/internal/gc.compileFunctions.func3.1()
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:166 +0x30 fp=0xc008c2bfe0 sp=0xc008c2bfb0 pc=0xc46d70
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc008c2bfe8 sp=0xc008c2bfe0 pc=0x46a441
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 24
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:165 +0x23a

goroutine 901 [runnable]:
cmd/internal/obj/x86.padJumpsCtx.reAssemble(0x20?, 0xc00023f480)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/internal/obj/x86/asm6.go:2025 +0xbf fp=0xc008a9bbe8 sp=0xc008a9bbe0 pc=0x6bfedf
cmd/internal/obj/x86.span6(0xc000186400, 0xc000a18380, 0xc0021cf8b0?)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/internal/obj/x86/asm6.go:2161 +0x9a5 fp=0xc008a9bd78 sp=0xc008a9bbe8 pc=0x6c08a5
cmd/internal/obj.Flushplist(0xc000186400, 0xc008a9bf40, 0xc0021cf8b0, {0x7ffea33c4841, 0x27})
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/internal/obj/plist.go:152 +0x849 fp=0xc008a9bea8 sp=0xc008a9bd78 pc=0x541ca9
cmd/compile/internal/objw.(*Progs).Flush(...)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/objw/prog.go:124
cmd/compile/internal/ssagen.Compile(0xc0034ed340, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/pgen.go:212 +0x4a6 fp=0xc008a9bf70 sp=0xc008a9bea8 pc=0xa5b7c6
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:184 +0x34 fp=0xc008a9bfb0 sp=0xc008a9bf70 pc=0xc46954
cmd/compile/internal/gc.compileFunctions.func3.1()
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:166 +0x30 fp=0xc008a9bfe0 sp=0xc008a9bfb0 pc=0xc46d70
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc008a9bfe8 sp=0xc008a9bfe0 pc=0x46a441
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 24
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:165 +0x23a

goroutine 914 [runnable]:
runtime.mapassign(0xcc1660?, 0xc008c8fb60?, 0xc008c8fe00?)
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/map.go:579 +0x525 fp=0xc008ca3c78 sp=0xc008ca3c70 pc=0x40e425
cmd/compile/internal/ssa.deadcode(0xc00478f520)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/deadcode.go:233 +0x1085 fp=0xc008ca3ec8 sp=0xc008ca3c78 pc=0x7012c5
cmd/compile/internal/ssa.Compile(0xc00478f520)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/compile.go:97 +0x95e fp=0xc008ca7b80 sp=0xc008ca3ec8 pc=0x6f81be
cmd/compile/internal/ssagen.buildssa(0xc0033dc000, 0x3)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/ssa.go:567 +0x1fc7 fp=0xc008ca7ea8 sp=0xc008ca7b80 pc=0xa63ee7
cmd/compile/internal/ssagen.Compile(0xc0033dc000, 0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/pgen.go:187 +0x45 fp=0xc008ca7f70 sp=0xc008ca7ea8 pc=0xa5b365
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:184 +0x34 fp=0xc008ca7fb0 sp=0xc008ca7f70 pc=0xc46954
cmd/compile/internal/gc.compileFunctions.func3.1()
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:166 +0x30 fp=0xc008ca7fe0 sp=0xc008ca7fb0 pc=0xc46d70
runtime.goexit()
	../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc008ca7fe8 sp=0xc008ca7fe0 pc=0x46a441
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 24
	../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:165 +0x23a

rax    0xa
rbx    0x7f957d0a7018
rcx    0x80
rdx    0x0
rdi    0x80
rsi    0xfa89481e75c98548
rbp    0xc000075ce8
rsp    0xc000075cb0
r8     0xc002444340
r9     0x7f957d0a7028
r10    0x7e
r11    0x48d3294803e2c148
r12    0x12
r13    0x1335d80
r14    0xc0000069c0
r15    0x3
rip    0x458f97
rflags 0x10286
cs     0x33
fs     0x0
gs     0x0
error: exit status 2
error: exit status 1

(This was run with a modified Go tree, but the only change was replacing cputicks with nanotime and removing the trace tick division in in src/runtime/trace.go.)

I don't think this is easily reproducible yet so I'll keep running and try to reproduce.

CC @felixge @nsrip-dd @prattmic

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Apr 18, 2023
@mknyszek mknyszek added this to the Go1.21 milestone Apr 18, 2023
@mknyszek mknyszek added release-blocker NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed compiler/runtime Issues related to the Go compiler and/or runtime. labels Apr 18, 2023
@bcmills bcmills added the compiler/runtime Issues related to the Go compiler and/or runtime. label Apr 19, 2023
@nsrip-dd
Copy link
Contributor

I can reproduce this on master (at 969ab34) for the TestDeferHeapAndStack test in the runtime package. It crashes pretty consistently, though not always at the same place. Here's the test crashing within a traceGoPreempt call during morestack:

go test -trace /dev/null -run TestDeferHeapAndStack
SIGSEGV: segmentation violation
PC=0x45ead7 m=4 sigcode=1

goroutine 0 [idle]:
runtime.fpTracebackPCs(...)
  /home/ec2-user/repos/go/src/runtime/trace.go:944
runtime.traceStackID(0x27237f6c979861?, {0x7fcc7c0da018, 0x0?, 0x80}, 0x0?)
  /home/ec2-user/repos/go/src/runtime/trace.go:916 +0x217 fp=0x7fcc7f5fdba0 sp=0x7fcc7f5fdb58 pc=0x45ead7
runtime.traceEventLocked(0x6800020516?, 0x43d04e?, 0x18bc4?, 0xc00003dbd0, 0x12, 0x0, 0x1, {0x0, 0x0, 0x0})
  /home/ec2-user/repos/go/src/runtime/trace.go:759 +0x285 fp=0x7fcc7f5fdc18 sp=0x7fcc7f5fdba0 pc=0x45e0a5
runtime.traceEvent(0x0?, 0x1, {0x0, 0x0, 0x0})
  /home/ec2-user/repos/go/src/runtime/trace.go:691 +0xa9 fp=0x7fcc7f5fdc80 sp=0x7fcc7f5fdc18 pc=0x45dda9
runtime.traceGoPreempt(...)
  /home/ec2-user/repos/go/src/runtime/trace.go:1534
runtime.gopreempt_m(0xc0002a2340?)
  /home/ec2-user/repos/go/src/runtime/proc.go:3550 +0x45 fp=0x7fcc7f5fdcc0 sp=0x7fcc7f5fdc80 pc=0x443b45
runtime.newstack()
  /home/ec2-user/repos/go/src/runtime/stack.go:1090 +0x3cb fp=0x7fcc7f5fde70 sp=0x7fcc7f5fdcc0 pc=0x45514b
runtime.morestack()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:593 +0x8f fp=0x7fcc7f5fde78 sp=0x7fcc7f5fde70 pc=0x47526f

goroutine 21 [running]:
runtime.gcMarkDone()
  /home/ec2-user/repos/go/src/runtime/mgc.go:783 +0x2f0 fp=0xc000550ec8 sp=0xc000550ec0 pc=0x41dbd0
runtime.gcAssistAlloc(0xc0002a2820)
  /home/ec2-user/repos/go/src/runtime/mgcmark.go:483 +0x1e7 fp=0xc000550f28 sp=0xc000550ec8 pc=0x4217e7
runtime.deductAssistCredit(0x8?)
  /home/ec2-user/repos/go/src/runtime/malloc.go:1224 +0x54 fp=0xc000550f50 sp=0xc000550f28 pc=0x40fcb4
runtime.mallocgc(0x8, 0x7a4ac0, 0x1)
  /home/ec2-user/repos/go/src/runtime/malloc.go:939 +0xc9 fp=0xc000550fb8 sp=0xc000550f50 pc=0x40f4e9
runtime.newobject(0xb6?)
  /home/ec2-user/repos/go/src/runtime/malloc.go:1261 +0x25 fp=0xc000550fe0 sp=0xc000550fb8 pc=0x40fda5
runtime_test.deferHeapAndStack(0xb7)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:903 +0x3d fp=0xc0005510c0 sp=0xc000550fe0 pc=0x75a3dd
runtime_test.deferHeapAndStack(0xb8)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005511a0 sp=0xc0005510c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb9)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551280 sp=0xc0005511a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xba)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551360 sp=0xc000551280 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbb)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551440 sp=0xc000551360 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbc)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551520 sp=0xc000551440 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbd)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551600 sp=0xc000551520 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbe)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005516e0 sp=0xc000551600 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbf)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005517c0 sp=0xc0005516e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc0)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005518a0 sp=0xc0005517c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc1)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551980 sp=0xc0005518a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc2)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551a60 sp=0xc000551980 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc3)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551b40 sp=0xc000551a60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc4)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551c20 sp=0xc000551b40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc5)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551d00 sp=0xc000551c20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc6)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551de0 sp=0xc000551d00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc7)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551ec0 sp=0xc000551de0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc8)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000551fa0 sp=0xc000551ec0 pc=0x75a4e9
runtime_test.TestDeferHeapAndStack.func1()
  /home/ec2-user/repos/go/src/runtime/stack_test.go:890 +0x5a fp=0xc000551fe0 sp=0xc000551fa0 pc=0x75a35a
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000551fe8 sp=0xc000551fe0 pc=0x4770c1
created by runtime_test.TestDeferHeapAndStack in goroutine 7
  /home/ec2-user/repos/go/src/runtime/stack_test.go:888 +0xf3

goroutine 1 [chan receive, locked to thread]:
runtime.gopark(0xc000151970?, 0x40fa25?, 0x68?, 0x21?, 0x18?)
  /home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc000151908 sp=0xc0001518e8 pc=0x43d04e
runtime.chanrecv(0xc0000741c0, 0xc0001519ef, 0x1)
  /home/ec2-user/repos/go/src/runtime/chan.go:583 +0x3cd fp=0xc000151980 sp=0xc000151908 pc=0x408bed
runtime.chanrecv1(0xaf4940?, 0x7a55c0?)
  /home/ec2-user/repos/go/src/runtime/chan.go:442 +0x12 fp=0xc0001519a8 sp=0xc000151980 pc=0x4087f2
testing.(*T).Run(0xc000160340, {0x835b27?, 0x4fee1c?}, 0x857958)
  /home/ec2-user/repos/go/src/testing/testing.go:1649 +0x3c8 fp=0xc000151a68 sp=0xc0001519a8 pc=0x5003a8
testing.runTests.func1(0xaf5800?)
  /home/ec2-user/repos/go/src/testing/testing.go:2055 +0x3e fp=0xc000151ab8 sp=0xc000151a68 pc=0x50247e
testing.tRunner(0xc000160340, 0xc000151bd0)
  /home/ec2-user/repos/go/src/testing/testing.go:1595 +0xff fp=0xc000151b08 sp=0xc000151ab8 pc=0x4ff53f
testing.runTests(0xc00011a6e0?, {0xaf1020, 0x19e, 0x19e}, {0x417b68?, 0xc00011f500?, 0xaf4e60?})
  /home/ec2-user/repos/go/src/testing/testing.go:2053 +0x445 fp=0xc000151c00 sp=0xc000151b08 pc=0x502365
testing.(*M).Run(0xc00011a6e0)
  /home/ec2-user/repos/go/src/testing/testing.go:1925 +0x636 fp=0xc000151e48 sp=0xc000151c00 pc=0x500d96
runtime_test.TestMain(0x476d5a?)
  /home/ec2-user/repos/go/src/runtime/crash_test.go:27 +0x13 fp=0xc000151e88 sp=0xc000151e48 pc=0x6da433
main.main()
  _testmain.go:1463 +0x1c6 fp=0xc000151f40 sp=0xc000151e88 pc=0x775da6
runtime.main()
  /home/ec2-user/repos/go/src/runtime/proc.go:255 +0x2b2 fp=0xc000151fe0 sp=0xc000151f40 pc=0x43cbf2
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000151fe8 sp=0xc000151fe0 pc=0x4770c1

goroutine 2 [force gc (idle)]:
runtime.gopark(0xaccb00?, 0xaf5800?, 0x0?, 0x0?, 0x0?)
  /home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00004a7a8 sp=0xc00004a788 pc=0x43d04e
runtime.goparkunlock(...)
  /home/ec2-user/repos/go/src/runtime/proc.go:392
runtime.forcegchelper()
  /home/ec2-user/repos/go/src/runtime/proc.go:310 +0xb3 fp=0xc00004a7e0 sp=0xc00004a7a8 pc=0x43ceb3
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00004a7e8 sp=0xc00004a7e0 pc=0x4770c1
created by runtime.init.6 in goroutine 1
  /home/ec2-user/repos/go/src/runtime/proc.go:298 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0xaf4a01?, 0x0?, 0x0?, 0x0?, 0x0?)
  /home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00004af78 sp=0xc00004af58 pc=0x43d04e
runtime.goparkunlock(...)
  /home/ec2-user/repos/go/src/runtime/proc.go:392
runtime.bgsweep(0x0?)
  /home/ec2-user/repos/go/src/runtime/mgcsweep.go:319 +0xdf fp=0xc00004afc8 sp=0xc00004af78 pc=0x427b9f
runtime.gcenable.func1()
  /home/ec2-user/repos/go/src/runtime/mgc.go:178 +0x25 fp=0xc00004afe0 sp=0xc00004afc8 pc=0x41cf05
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00004afe8 sp=0xc00004afe0 pc=0x4770c1
created by runtime.gcenable in goroutine 1
  /home/ec2-user/repos/go/src/runtime/mgc.go:178 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x11a4c?, 0x3b9aca00?, 0x0?, 0x0?, 0x0?)
  /home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00004b770 sp=0xc00004b750 pc=0x43d04e
runtime.goparkunlock(...)
  /home/ec2-user/repos/go/src/runtime/proc.go:392
runtime.(*scavengerState).park(0xaf4ee0)
  /home/ec2-user/repos/go/src/runtime/mgcscavenge.go:399 +0x49 fp=0xc00004b7a0 sp=0xc00004b770 pc=0x425b29
runtime.bgscavenge(0x0?)
  /home/ec2-user/repos/go/src/runtime/mgcscavenge.go:632 +0x59 fp=0xc00004b7c8 sp=0xc00004b7a0 pc=0x4260d9
runtime.gcenable.func2()
  /home/ec2-user/repos/go/src/runtime/mgc.go:179 +0x25 fp=0xc00004b7e0 sp=0xc00004b7c8 pc=0x41cea5
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00004b7e8 sp=0xc00004b7e0 pc=0x4770c1
created by runtime.gcenable in goroutine 1
  /home/ec2-user/repos/go/src/runtime/mgc.go:179 +0xa5

goroutine 5 [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
  /home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00004be28 sp=0xc00004be08 pc=0x43d04e
runtime.runfinq()
  /home/ec2-user/repos/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00004bfe0 sp=0xc00004be28 pc=0x41bf27
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00004bfe8 sp=0xc00004bfe0 pc=0x4770c1
created by runtime.createfing in goroutine 1
  /home/ec2-user/repos/go/src/runtime/mfinal.go:163 +0x3d

goroutine 6 [trace reader (blocked)]:
runtime.gopark(0xc00004c778?, 0xc0000245a0?, 0x18?, 0xc4?, 0xfbb6?)
  /home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00004c738 sp=0xc00004c718 pc=0x43d04e
runtime.ReadTrace()
  /home/ec2-user/repos/go/src/runtime/trace.go:445 +0x31 fp=0xc00004c7a0 sp=0xc00004c738 pc=0x45d4f1
runtime/trace.Start.func1()
  /home/ec2-user/repos/go/src/runtime/trace/trace.go:130 +0x45 fp=0xc00004c7e0 sp=0xc00004c7a0 pc=0x4eeca5
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00004c7e8 sp=0xc00004c7e0 pc=0x4770c1
created by runtime/trace.Start in goroutine 1
  /home/ec2-user/repos/go/src/runtime/trace/trace.go:128 +0xdf

goroutine 7 [chan receive]:
runtime.gopark(0xc00004cee8?, 0x40fa25?, 0x60?, 0x40?, 0x20?)
  /home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00004ce80 sp=0xc00004ce60 pc=0x43d04e
runtime.chanrecv(0xc0002b2000, 0x0, 0x1)
  /home/ec2-user/repos/go/src/runtime/chan.go:583 +0x3cd fp=0xc00004cef8 sp=0xc00004ce80 pc=0x408bed
runtime.chanrecv1(0xad3440?, 0x0?)
  /home/ec2-user/repos/go/src/runtime/chan.go:442 +0x12 fp=0xc00004cf20 sp=0xc00004cef8 pc=0x4087f2
runtime_test.TestDeferHeapAndStack(0x0?)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:898 +0x172 fp=0xc00004cf70 sp=0xc00004cf20 pc=0x75a292
testing.tRunner(0xc0001604e0, 0x857958)
  /home/ec2-user/repos/go/src/testing/testing.go:1595 +0xff fp=0xc00004cfc0 sp=0xc00004cf70 pc=0x4ff53f
testing.(*T).Run.func1()
  /home/ec2-user/repos/go/src/testing/testing.go:1648 +0x25 fp=0xc00004cfe0 sp=0xc00004cfc0 pc=0x500505
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00004cfe8 sp=0xc00004cfe0 pc=0x4770c1
created by testing.(*T).Run in goroutine 1
  /home/ec2-user/repos/go/src/testing/testing.go:1648 +0x3ad

goroutine 18 [GC mark termination]:
runtime.systemstack_switch()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc0002c34c8 sp=0xc0002c34b8 pc=0x475128
runtime.gcMarkDone()
  /home/ec2-user/repos/go/src/runtime/mgc.go:806 +0xff fp=0xc0002c3528 sp=0xc0002c34c8 pc=0x41d9df
runtime.gcAssistAlloc(0xc0002a2340)
  /home/ec2-user/repos/go/src/runtime/mgcmark.go:483 +0x1e7 fp=0xc0002c3588 sp=0xc0002c3528 pc=0x4217e7
runtime.deductAssistCredit(0x0?)
  /home/ec2-user/repos/go/src/runtime/malloc.go:1224 +0x54 fp=0xc0002c35b0 sp=0xc0002c3588 pc=0x40fcb4
runtime.mallocgc(0x400, 0x7a6640, 0x1)
  /home/ec2-user/repos/go/src/runtime/malloc.go:939 +0xc9 fp=0xc0002c3618 sp=0xc0002c35b0 pc=0x40f4e9
runtime.newobject(0x2b?)
  /home/ec2-user/repos/go/src/runtime/malloc.go:1261 +0x25 fp=0xc0002c3640 sp=0xc0002c3618 pc=0x40fda5
runtime_test.deferHeapAndStack(0x2c)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:924 +0x165 fp=0xc0002c3720 sp=0xc0002c3640 pc=0x75a505
runtime_test.deferHeapAndStack(0x2d)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c3800 sp=0xc0002c3720 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x2e)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c38e0 sp=0xc0002c3800 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x2f)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c39c0 sp=0xc0002c38e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x30)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c3aa0 sp=0xc0002c39c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x31)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c3b80 sp=0xc0002c3aa0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x32)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c3c60 sp=0xc0002c3b80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x33)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c3d40 sp=0xc0002c3c60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x34)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c3e20 sp=0xc0002c3d40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x35)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c3f00 sp=0xc0002c3e20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x36)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c3fe0 sp=0xc0002c3f00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x37)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c40c0 sp=0xc0002c3fe0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x38)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c41a0 sp=0xc0002c40c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x39)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4280 sp=0xc0002c41a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3a)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4360 sp=0xc0002c4280 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3b)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4440 sp=0xc0002c4360 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3c)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4520 sp=0xc0002c4440 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3d)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4600 sp=0xc0002c4520 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3e)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c46e0 sp=0xc0002c4600 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3f)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c47c0 sp=0xc0002c46e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x40)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c48a0 sp=0xc0002c47c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x41)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4980 sp=0xc0002c48a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x42)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4a60 sp=0xc0002c4980 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x43)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4b40 sp=0xc0002c4a60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x44)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4c20 sp=0xc0002c4b40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x45)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4d00 sp=0xc0002c4c20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x46)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4de0 sp=0xc0002c4d00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x47)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4ec0 sp=0xc0002c4de0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x48)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c4fa0 sp=0xc0002c4ec0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x49)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5080 sp=0xc0002c4fa0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4a)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5160 sp=0xc0002c5080 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4b)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5240 sp=0xc0002c5160 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4c)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5320 sp=0xc0002c5240 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4d)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5400 sp=0xc0002c5320 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4e)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c54e0 sp=0xc0002c5400 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4f)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c55c0 sp=0xc0002c54e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x50)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c56a0 sp=0xc0002c55c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x51)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5780 sp=0xc0002c56a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x52)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5860 sp=0xc0002c5780 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x53)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5940 sp=0xc0002c5860 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x54)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5a20 sp=0xc0002c5940 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x55)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5b00 sp=0xc0002c5a20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x56)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5be0 sp=0xc0002c5b00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x57)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c5cc0 sp=0xc0002c5be0 pc=0x75a4e9
...65 frames elided...
runtime_test.deferHeapAndStack(0x99)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c9680 sp=0xc0002c95a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9a)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c9760 sp=0xc0002c9680 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9b)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c9840 sp=0xc0002c9760 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9c)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c9920 sp=0xc0002c9840 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9d)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c9a00 sp=0xc0002c9920 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9e)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c9ae0 sp=0xc0002c9a00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9f)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c9bc0 sp=0xc0002c9ae0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa0)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c9ca0 sp=0xc0002c9bc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa1)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c9d80 sp=0xc0002c9ca0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa2)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c9e60 sp=0xc0002c9d80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa3)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002c9f40 sp=0xc0002c9e60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa4)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca020 sp=0xc0002c9f40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa5)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca100 sp=0xc0002ca020 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa6)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca1e0 sp=0xc0002ca100 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa7)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca2c0 sp=0xc0002ca1e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa8)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca3a0 sp=0xc0002ca2c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa9)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca480 sp=0xc0002ca3a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaa)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca560 sp=0xc0002ca480 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xab)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca640 sp=0xc0002ca560 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xac)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca720 sp=0xc0002ca640 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xad)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca800 sp=0xc0002ca720 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xae)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca8e0 sp=0xc0002ca800 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaf)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002ca9c0 sp=0xc0002ca8e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb0)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002caaa0 sp=0xc0002ca9c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb1)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cab80 sp=0xc0002caaa0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb2)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cac60 sp=0xc0002cab80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb3)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cad40 sp=0xc0002cac60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb4)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cae20 sp=0xc0002cad40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb5)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002caf00 sp=0xc0002cae20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb6)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cafe0 sp=0xc0002caf00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb7)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cb0c0 sp=0xc0002cafe0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb8)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cb1a0 sp=0xc0002cb0c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb9)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cb280 sp=0xc0002cb1a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xba)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cb360 sp=0xc0002cb280 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbb)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cb440 sp=0xc0002cb360 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbc)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cb520 sp=0xc0002cb440 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbd)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cb600 sp=0xc0002cb520 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbe)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cb6e0 sp=0xc0002cb600 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbf)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cb7c0 sp=0xc0002cb6e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc0)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cb8a0 sp=0xc0002cb7c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc1)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cb980 sp=0xc0002cb8a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc2)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cba60 sp=0xc0002cb980 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc3)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cbb40 sp=0xc0002cba60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc4)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cbc20 sp=0xc0002cbb40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc5)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cbd00 sp=0xc0002cbc20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc6)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cbde0 sp=0xc0002cbd00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc7)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cbec0 sp=0xc0002cbde0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc8)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0002cbfa0 sp=0xc0002cbec0 pc=0x75a4e9
runtime_test.TestDeferHeapAndStack.func1()
  /home/ec2-user/repos/go/src/runtime/stack_test.go:890 +0x5a fp=0xc0002cbfe0 sp=0xc0002cbfa0 pc=0x75a35a
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0002cbfe8 sp=0xc0002cbfe0 pc=0x4770c1
created by runtime_test.TestDeferHeapAndStack in goroutine 7
  /home/ec2-user/repos/go/src/runtime/stack_test.go:888 +0xf3

goroutine 19 [runnable]:
runtime.deferproc(0xc0002bae80?)
  /home/ec2-user/repos/go/src/runtime/panic.go:270 +0xf2 fp=0xc000522440 sp=0xc000522438 pc=0x4387f2
runtime_test.deferHeapAndStack(0x3c)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:910 +0x1fe fp=0xc000522520 sp=0xc000522440 pc=0x75a59e
runtime_test.deferHeapAndStack(0x3d)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000522600 sp=0xc000522520 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3e)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005226e0 sp=0xc000522600 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3f)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005227c0 sp=0xc0005226e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x40)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005228a0 sp=0xc0005227c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x41)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000522980 sp=0xc0005228a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x42)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000522a60 sp=0xc000522980 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x43)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000522b40 sp=0xc000522a60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x44)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000522c20 sp=0xc000522b40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x45)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000522d00 sp=0xc000522c20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x46)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000522de0 sp=0xc000522d00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x47)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000522ec0 sp=0xc000522de0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x48)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000522fa0 sp=0xc000522ec0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x49)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523080 sp=0xc000522fa0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4a)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523160 sp=0xc000523080 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4b)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523240 sp=0xc000523160 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4c)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523320 sp=0xc000523240 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4d)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523400 sp=0xc000523320 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4e)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005234e0 sp=0xc000523400 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4f)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005235c0 sp=0xc0005234e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x50)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005236a0 sp=0xc0005235c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x51)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523780 sp=0xc0005236a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x52)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523860 sp=0xc000523780 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x53)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523940 sp=0xc000523860 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x54)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523a20 sp=0xc000523940 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x55)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523b00 sp=0xc000523a20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x56)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523be0 sp=0xc000523b00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x57)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523cc0 sp=0xc000523be0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x58)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523da0 sp=0xc000523cc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x59)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523e80 sp=0xc000523da0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5a)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000523f60 sp=0xc000523e80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5b)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524040 sp=0xc000523f60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5c)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524120 sp=0xc000524040 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5d)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524200 sp=0xc000524120 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5e)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005242e0 sp=0xc000524200 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5f)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005243c0 sp=0xc0005242e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x60)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005244a0 sp=0xc0005243c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x61)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524580 sp=0xc0005244a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x62)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524660 sp=0xc000524580 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x63)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524740 sp=0xc000524660 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x64)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524820 sp=0xc000524740 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x65)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524900 sp=0xc000524820 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x66)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005249e0 sp=0xc000524900 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x67)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524ac0 sp=0xc0005249e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x68)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524ba0 sp=0xc000524ac0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x69)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524c80 sp=0xc000524ba0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x6a)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524d60 sp=0xc000524c80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x6b)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524e40 sp=0xc000524d60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x6c)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000524f20 sp=0xc000524e40 pc=0x75a4e9
...44 frames elided...
runtime_test.deferHeapAndStack(0x99)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000527680 sp=0xc0005275a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9a)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000527760 sp=0xc000527680 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9b)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000527840 sp=0xc000527760 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9c)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000527920 sp=0xc000527840 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9d)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000527a00 sp=0xc000527920 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9e)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000527ae0 sp=0xc000527a00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9f)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000527bc0 sp=0xc000527ae0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa0)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000527ca0 sp=0xc000527bc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa1)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000527d80 sp=0xc000527ca0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa2)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000527e60 sp=0xc000527d80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa3)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000527f40 sp=0xc000527e60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa4)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528020 sp=0xc000527f40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa5)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528100 sp=0xc000528020 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa6)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005281e0 sp=0xc000528100 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa7)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005282c0 sp=0xc0005281e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa8)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005283a0 sp=0xc0005282c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa9)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528480 sp=0xc0005283a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaa)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528560 sp=0xc000528480 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xab)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528640 sp=0xc000528560 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xac)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528720 sp=0xc000528640 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xad)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528800 sp=0xc000528720 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xae)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005288e0 sp=0xc000528800 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaf)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005289c0 sp=0xc0005288e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb0)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528aa0 sp=0xc0005289c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb1)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528b80 sp=0xc000528aa0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb2)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528c60 sp=0xc000528b80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb3)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528d40 sp=0xc000528c60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb4)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528e20 sp=0xc000528d40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb5)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528f00 sp=0xc000528e20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb6)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000528fe0 sp=0xc000528f00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb7)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005290c0 sp=0xc000528fe0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb8)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005291a0 sp=0xc0005290c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb9)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529280 sp=0xc0005291a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xba)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529360 sp=0xc000529280 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbb)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529440 sp=0xc000529360 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbc)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529520 sp=0xc000529440 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbd)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529600 sp=0xc000529520 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbe)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005296e0 sp=0xc000529600 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbf)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005297c0 sp=0xc0005296e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc0)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0005298a0 sp=0xc0005297c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc1)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529980 sp=0xc0005298a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc2)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529a60 sp=0xc000529980 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc3)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529b40 sp=0xc000529a60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc4)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529c20 sp=0xc000529b40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc5)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529d00 sp=0xc000529c20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc6)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529de0 sp=0xc000529d00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc7)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529ec0 sp=0xc000529de0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc8)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000529fa0 sp=0xc000529ec0 pc=0x75a4e9
runtime_test.TestDeferHeapAndStack.func1()
  /home/ec2-user/repos/go/src/runtime/stack_test.go:890 +0x5a fp=0xc000529fe0 sp=0xc000529fa0 pc=0x75a35a
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000529fe8 sp=0xc000529fe0 pc=0x4770c1
created by runtime_test.TestDeferHeapAndStack in goroutine 7
  /home/ec2-user/repos/go/src/runtime/stack_test.go:888 +0xf3

goroutine 20 [runnable]:
runtime.deferproc(0xc00031ace0?)
  /home/ec2-user/repos/go/src/runtime/panic.go:270 +0xf2 fp=0xc00045e340 sp=0xc00045e338 pc=0x4387f2
runtime_test.deferHeapAndStack(0x84)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:910 +0x1fe fp=0xc00045e420 sp=0xc00045e340 pc=0x75a59e
runtime_test.deferHeapAndStack(0x85)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045e500 sp=0xc00045e420 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x86)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045e5e0 sp=0xc00045e500 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x87)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045e6c0 sp=0xc00045e5e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x88)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045e7a0 sp=0xc00045e6c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x89)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045e880 sp=0xc00045e7a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8a)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045e960 sp=0xc00045e880 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8b)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045ea40 sp=0xc00045e960 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8c)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045eb20 sp=0xc00045ea40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8d)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045ec00 sp=0xc00045eb20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8e)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045ece0 sp=0xc00045ec00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8f)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045edc0 sp=0xc00045ece0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x90)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045eea0 sp=0xc00045edc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x91)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045ef80 sp=0xc00045eea0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x92)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045f060 sp=0xc00045ef80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x93)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045f140 sp=0xc00045f060 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x94)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045f220 sp=0xc00045f140 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x95)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045f300 sp=0xc00045f220 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x96)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045f3e0 sp=0xc00045f300 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x97)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045f4c0 sp=0xc00045f3e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x98)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045f5a0 sp=0xc00045f4c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x99)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045f680 sp=0xc00045f5a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9a)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045f760 sp=0xc00045f680 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9b)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045f840 sp=0xc00045f760 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9c)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045f920 sp=0xc00045f840 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9d)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045fa00 sp=0xc00045f920 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9e)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045fae0 sp=0xc00045fa00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9f)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045fbc0 sp=0xc00045fae0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa0)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045fca0 sp=0xc00045fbc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa1)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045fd80 sp=0xc00045fca0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa2)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045fe60 sp=0xc00045fd80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa3)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00045ff40 sp=0xc00045fe60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa4)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460020 sp=0xc00045ff40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa5)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460100 sp=0xc000460020 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa6)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004601e0 sp=0xc000460100 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa7)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004602c0 sp=0xc0004601e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa8)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004603a0 sp=0xc0004602c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa9)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460480 sp=0xc0004603a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaa)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460560 sp=0xc000460480 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xab)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460640 sp=0xc000460560 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xac)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460720 sp=0xc000460640 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xad)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460800 sp=0xc000460720 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xae)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004608e0 sp=0xc000460800 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaf)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004609c0 sp=0xc0004608e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb0)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460aa0 sp=0xc0004609c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb1)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460b80 sp=0xc000460aa0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb2)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460c60 sp=0xc000460b80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb3)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460d40 sp=0xc000460c60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb4)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460e20 sp=0xc000460d40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb5)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460f00 sp=0xc000460e20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb6)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000460fe0 sp=0xc000460f00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb7)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004610c0 sp=0xc000460fe0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb8)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004611a0 sp=0xc0004610c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb9)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461280 sp=0xc0004611a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xba)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461360 sp=0xc000461280 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbb)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461440 sp=0xc000461360 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbc)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461520 sp=0xc000461440 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbd)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461600 sp=0xc000461520 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbe)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004616e0 sp=0xc000461600 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbf)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004617c0 sp=0xc0004616e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc0)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004618a0 sp=0xc0004617c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc1)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461980 sp=0xc0004618a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc2)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461a60 sp=0xc000461980 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc3)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461b40 sp=0xc000461a60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc4)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461c20 sp=0xc000461b40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc5)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461d00 sp=0xc000461c20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc6)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461de0 sp=0xc000461d00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc7)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461ec0 sp=0xc000461de0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc8)
  /home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000461fa0 sp=0xc000461ec0 pc=0x75a4e9
runtime_test.TestDeferHeapAndStack.func1()
  /home/ec2-user/repos/go/src/runtime/stack_test.go:890 +0x5a fp=0xc000461fe0 sp=0xc000461fa0 pc=0x75a35a
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000461fe8 sp=0xc000461fe0 pc=0x4770c1
created by runtime_test.TestDeferHeapAndStack in goroutine 7
  /home/ec2-user/repos/go/src/runtime/stack_test.go:888 +0xf3

goroutine 34 [GC worker (idle)]:
runtime.gopark(0xd7eff01424013?, 0xc00002a1a0?, 0x1a?, 0x14?, 0x0?)
  /home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc000046f50 sp=0xc000046f30 pc=0x43d04e
runtime.gcBgMarkWorker()
  /home/ec2-user/repos/go/src/runtime/mgc.go:1286 +0xe5 fp=0xc000046fe0 sp=0xc000046f50 pc=0x41eb45
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000046fe8 sp=0xc000046fe0 pc=0x4770c1
created by runtime.gcBgMarkStartWorkers in goroutine 19
  /home/ec2-user/repos/go/src/runtime/mgc.go:1210 +0x1c

goroutine 22 [GC worker (idle)]:
runtime.gopark(0xd7eff0142602d?, 0x1?, 0x77?, 0x42?, 0x0?)
  /home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc000047f50 sp=0xc000047f30 pc=0x43d04e
runtime.gcBgMarkWorker()
  /home/ec2-user/repos/go/src/runtime/mgc.go:1286 +0xe5 fp=0xc000047fe0 sp=0xc000047f50 pc=0x41eb45
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x4770c1
created by runtime.gcBgMarkStartWorkers in goroutine 19
  /home/ec2-user/repos/go/src/runtime/mgc.go:1210 +0x1c

goroutine 50 [GC worker (idle)]:
runtime.gopark(0xd7eff00c48380?, 0x3?, 0xcc?, 0x4f?, 0x0?)
  /home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc000047750 sp=0xc000047730 pc=0x43d04e
runtime.gcBgMarkWorker()
  /home/ec2-user/repos/go/src/runtime/mgc.go:1286 +0xe5 fp=0xc0000477e0 sp=0xc000047750 pc=0x41eb45
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0000477e8 sp=0xc0000477e0 pc=0x4770c1
created by runtime.gcBgMarkStartWorkers in goroutine 19
  /home/ec2-user/repos/go/src/runtime/mgc.go:1210 +0x1c

goroutine 35 [GC worker (idle)]:
runtime.gopark(0xd7eff000310ec?, 0x3?, 0xc0?, 0x4b?, 0x0?)
  /home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00053a750 sp=0xc00053a730 pc=0x43d04e
runtime.gcBgMarkWorker()
  /home/ec2-user/repos/go/src/runtime/mgc.go:1286 +0xe5 fp=0xc00053a7e0 sp=0xc00053a750 pc=0x41eb45
runtime.goexit()
  /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00053a7e8 sp=0xc00053a7e0 pc=0x4770c1
created by runtime.gcBgMarkStartWorkers in goroutine 19
  /home/ec2-user/repos/go/src/runtime/mgc.go:1210 +0x1c

rax    0x1
rbx    0x7fcc7c0da018
rcx    0x80
rdx    0x0
rdi    0x80
rsi    0x1
rbp    0x7fcc7f5fdb90
rsp    0x7fcc7f5fdb58
r8     0xc0002a2820
r9     0x7fcc7c0da028
r10    0x7e
r11    0xc000080400
r12    0x12
r13    0x0
r14    0xc000007520
r15    0x1
rip    0x45ead7
rflags 0x10202
cs     0x33
fs     0x0
gs     0x0
exit status 2
FAIL	runtime	0.742s

The test does not always crash at the same place:

Another example
SIGSEGV: segmentation violation
PC=0x45ead7 m=5 sigcode=1

goroutine 0 [idle]:
runtime.fpTracebackPCs(...)
	/home/ec2-user/repos/go/src/runtime/trace.go:944
runtime.traceStackID(0x2721efd9baacfd?, {0x7fafc0092018, 0xc000007d40?, 0x80}, 0xc000040138?)
	/home/ec2-user/repos/go/src/runtime/trace.go:916 +0x217 fp=0x7fafbb5fdcc8 sp=0x7fafbb5fdc80 pc=0x45ead7
runtime.traceEventLocked(0xaf60f0?, 0x80?, 0xbb5fdd88?, 0xc0000400d0, 0x15, 0x0, 0x5, {0x7fafbb5fddd8, 0x2, 0x2})
	/home/ec2-user/repos/go/src/runtime/trace.go:759 +0x285 fp=0x7fafbb5fdd40 sp=0x7fafbb5fdcc8 pc=0x45e0a5
runtime.traceEvent(0x55?, 0x5, {0x7fafbb5fddd8, 0x2, 0x2})
	/home/ec2-user/repos/go/src/runtime/trace.go:691 +0xa9 fp=0x7fafbb5fdda8 sp=0x7fafbb5fdd40 pc=0x45dda9
runtime.traceGoUnpark(0x7fafbb5fde60?, 0x43f436?)
	/home/ec2-user/repos/go/src/runtime/trace.go:1551 +0x97 fp=0x7fafbb5fddf8 sp=0x7fafbb5fdda8 pc=0x460a17
runtime.ready(0xc0005201a0, 0xc000036000?, 0x17?)
	/home/ec2-user/repos/go/src/runtime/proc.go:873 +0x29 fp=0x7fafbb5fde48 sp=0x7fafbb5fddf8 pc=0x43e349
runtime.goready.func1()
	/home/ec2-user/repos/go/src/runtime/proc.go:397 +0x25 fp=0x7fafbb5fde70 sp=0x7fafbb5fde48 pc=0x43d165
runtime.systemstack()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:509 +0x4a fp=0x7fafbb5fde80 sp=0x7fafbb5fde70 pc=0x47518a

goroutine 7 [running]:
runtime.systemstack_switch()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc000164740 sp=0xc000164730 pc=0x475128
runtime.goready(0xc0001647d8?, 0xc00051c3c0?)
	/home/ec2-user/repos/go/src/runtime/proc.go:396 +0x45 fp=0xc000164770 sp=0xc000164740 pc=0x43d105
runtime.readyWithTime(0xafae80?, 0xc000080000?)
	/home/ec2-user/repos/go/src/runtime/sema.go:99 +0x4f fp=0xc000164798 sp=0xc000164770 pc=0x44de4f
runtime.semrelease1(0xc000164828?, 0x0, 0x0)
	/home/ec2-user/repos/go/src/runtime/sema.go:210 +0x169 fp=0xc0001647e8 sp=0xc000164798 pc=0x44e369
runtime.semrelease(...)
	/home/ec2-user/repos/go/src/runtime/sema.go:172
runtime.gcMarkDone()
	/home/ec2-user/repos/go/src/runtime/mgc.go:905 +0x296 fp=0xc000164848 sp=0xc0001647e8 pc=0x41db76
runtime.gcAssistAlloc(0xc000007d40)
	/home/ec2-user/repos/go/src/runtime/mgcmark.go:483 +0x1e7 fp=0xc0001648a8 sp=0xc000164848 pc=0x4217e7
runtime.deductAssistCredit(0x416b8f?)
	/home/ec2-user/repos/go/src/runtime/malloc.go:1224 +0x54 fp=0xc0001648d0 sp=0xc0001648a8 pc=0x40fcb4
runtime.mallocgc(0x8, 0x7a4ac0, 0x1)
	/home/ec2-user/repos/go/src/runtime/malloc.go:939 +0xc9 fp=0xc000164938 sp=0xc0001648d0 pc=0x40f4e9
runtime.newobject(0x8a?)
	/home/ec2-user/repos/go/src/runtime/malloc.go:1261 +0x25 fp=0xc000164960 sp=0xc000164938 pc=0x40fda5
runtime_test.deferHeapAndStack(0x8b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:903 +0x3d fp=0xc000164a40 sp=0xc000164960 pc=0x75a3dd
runtime_test.deferHeapAndStack(0x8c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000164b20 sp=0xc000164a40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000164c00 sp=0xc000164b20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000164ce0 sp=0xc000164c00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000164dc0 sp=0xc000164ce0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x90)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000164ea0 sp=0xc000164dc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x91)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000164f80 sp=0xc000164ea0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x92)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165060 sp=0xc000164f80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x93)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165140 sp=0xc000165060 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x94)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165220 sp=0xc000165140 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x95)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165300 sp=0xc000165220 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x96)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001653e0 sp=0xc000165300 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x97)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001654c0 sp=0xc0001653e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x98)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001655a0 sp=0xc0001654c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x99)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165680 sp=0xc0001655a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9a)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165760 sp=0xc000165680 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165840 sp=0xc000165760 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165920 sp=0xc000165840 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165a00 sp=0xc000165920 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165ae0 sp=0xc000165a00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165bc0 sp=0xc000165ae0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165ca0 sp=0xc000165bc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165d80 sp=0xc000165ca0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165e60 sp=0xc000165d80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000165f40 sp=0xc000165e60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166020 sp=0xc000165f40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166100 sp=0xc000166020 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001661e0 sp=0xc000166100 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001662c0 sp=0xc0001661e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001663a0 sp=0xc0001662c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa9)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166480 sp=0xc0001663a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaa)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166560 sp=0xc000166480 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xab)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166640 sp=0xc000166560 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xac)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166720 sp=0xc000166640 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xad)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166800 sp=0xc000166720 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xae)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001668e0 sp=0xc000166800 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaf)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001669c0 sp=0xc0001668e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166aa0 sp=0xc0001669c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166b80 sp=0xc000166aa0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166c60 sp=0xc000166b80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166d40 sp=0xc000166c60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166e20 sp=0xc000166d40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166f00 sp=0xc000166e20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000166fe0 sp=0xc000166f00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001670c0 sp=0xc000166fe0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001671a0 sp=0xc0001670c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb9)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167280 sp=0xc0001671a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xba)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167360 sp=0xc000167280 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbb)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167440 sp=0xc000167360 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbc)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167520 sp=0xc000167440 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbd)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167600 sp=0xc000167520 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbe)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001676e0 sp=0xc000167600 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbf)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001677c0 sp=0xc0001676e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001678a0 sp=0xc0001677c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167980 sp=0xc0001678a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167a60 sp=0xc000167980 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167b40 sp=0xc000167a60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167c20 sp=0xc000167b40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167d00 sp=0xc000167c20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167de0 sp=0xc000167d00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167ec0 sp=0xc000167de0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000167fa0 sp=0xc000167ec0 pc=0x75a4e9
runtime_test.TestDeferHeapAndStack.func1()
	/home/ec2-user/repos/go/src/runtime/stack_test.go:890 +0x5a fp=0xc000167fe0 sp=0xc000167fa0 pc=0x75a35a
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000167fe8 sp=0xc000167fe0 pc=0x4770c1
created by runtime_test.TestDeferHeapAndStack in goroutine 20
	/home/ec2-user/repos/go/src/runtime/stack_test.go:888 +0xf3

goroutine 1 [chan receive, locked to thread]:
runtime.gopark(0xc0000f7970?, 0x40fa25?, 0x50?, 0x21?, 0x18?)
	/home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc0000f7908 sp=0xc0000f78e8 pc=0x43d04e
runtime.chanrecv(0xc000184150, 0xc0000f79ef, 0x1)
	/home/ec2-user/repos/go/src/runtime/chan.go:583 +0x3cd fp=0xc0000f7980 sp=0xc0000f7908 pc=0x408bed
runtime.chanrecv1(0xaf4940?, 0x7a55c0?)
	/home/ec2-user/repos/go/src/runtime/chan.go:442 +0x12 fp=0xc0000f79a8 sp=0xc0000f7980 pc=0x4087f2
testing.(*T).Run(0xc000082ea0, {0x835b27?, 0x4fee1c?}, 0x857958)
	/home/ec2-user/repos/go/src/testing/testing.go:1649 +0x3c8 fp=0xc0000f7a68 sp=0xc0000f79a8 pc=0x5003a8
testing.runTests.func1(0xaf5800?)
	/home/ec2-user/repos/go/src/testing/testing.go:2055 +0x3e fp=0xc0000f7ab8 sp=0xc0000f7a68 pc=0x50247e
testing.tRunner(0xc000082ea0, 0xc0000f7bd0)
	/home/ec2-user/repos/go/src/testing/testing.go:1595 +0xff fp=0xc0000f7b08 sp=0xc0000f7ab8 pc=0x4ff53f
testing.runTests(0xc0000ae780?, {0xaf1020, 0x19e, 0x19e}, {0x417b68?, 0xc0000aca90?, 0xaf4e60?})
	/home/ec2-user/repos/go/src/testing/testing.go:2053 +0x445 fp=0xc0000f7c00 sp=0xc0000f7b08 pc=0x502365
testing.(*M).Run(0xc0000ae780)
	/home/ec2-user/repos/go/src/testing/testing.go:1925 +0x636 fp=0xc0000f7e48 sp=0xc0000f7c00 pc=0x500d96
runtime_test.TestMain(0x476d5a?)
	/home/ec2-user/repos/go/src/runtime/crash_test.go:27 +0x13 fp=0xc0000f7e88 sp=0xc0000f7e48 pc=0x6da433
main.main()
	_testmain.go:1463 +0x1c6 fp=0xc0000f7f40 sp=0xc0000f7e88 pc=0x775da6
runtime.main()
	/home/ec2-user/repos/go/src/runtime/proc.go:255 +0x2b2 fp=0xc0000f7fe0 sp=0xc0000f7f40 pc=0x43cbf2
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x4770c1

goroutine 2 [force gc (idle)]:
runtime.gopark(0xaccb00?, 0xaf5800?, 0x0?, 0x0?, 0x0?)
	/home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00004a7a8 sp=0xc00004a788 pc=0x43d04e
runtime.goparkunlock(...)
	/home/ec2-user/repos/go/src/runtime/proc.go:392
runtime.forcegchelper()
	/home/ec2-user/repos/go/src/runtime/proc.go:310 +0xb3 fp=0xc00004a7e0 sp=0xc00004a7a8 pc=0x43ceb3
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00004a7e8 sp=0xc00004a7e0 pc=0x4770c1
created by runtime.init.6 in goroutine 1
	/home/ec2-user/repos/go/src/runtime/proc.go:298 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00004af78 sp=0xc00004af58 pc=0x43d04e
runtime.goparkunlock(...)
	/home/ec2-user/repos/go/src/runtime/proc.go:392
runtime.bgsweep(0x0?)
	/home/ec2-user/repos/go/src/runtime/mgcsweep.go:319 +0xdf fp=0xc00004afc8 sp=0xc00004af78 pc=0x427b9f
runtime.gcenable.func1()
	/home/ec2-user/repos/go/src/runtime/mgc.go:178 +0x25 fp=0xc00004afe0 sp=0xc00004afc8 pc=0x41cf05
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00004afe8 sp=0xc00004afe0 pc=0x4770c1
created by runtime.gcenable in goroutine 1
	/home/ec2-user/repos/go/src/runtime/mgc.go:178 +0x66

goroutine 4 [sleep]:
runtime.gopark(0xc000100000?, 0xd7e7574281eda?, 0x0?, 0x0?, 0x856270?)
	/home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00004b730 sp=0xc00004b710 pc=0x43d04e
runtime.goparkunlock(...)
	/home/ec2-user/repos/go/src/runtime/proc.go:392
runtime.(*scavengerState).sleep(0xaf4ee0, 0x4101008000000000)
	/home/ec2-user/repos/go/src/runtime/mgcscavenge.go:478 +0x113 fp=0xc00004b7a0 sp=0xc00004b730 pc=0x425cf3
runtime.bgscavenge(0x0?)
	/home/ec2-user/repos/go/src/runtime/mgcscavenge.go:636 +0x73 fp=0xc00004b7c8 sp=0xc00004b7a0 pc=0x4260f3
runtime.gcenable.func2()
	/home/ec2-user/repos/go/src/runtime/mgc.go:179 +0x25 fp=0xc00004b7e0 sp=0xc00004b7c8 pc=0x41cea5
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00004b7e8 sp=0xc00004b7e0 pc=0x4770c1
created by runtime.gcenable in goroutine 1
	/home/ec2-user/repos/go/src/runtime/mgc.go:179 +0xa5

goroutine 18 [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc000046628 sp=0xc000046608 pc=0x43d04e
runtime.runfinq()
	/home/ec2-user/repos/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000467e0 sp=0xc000046628 pc=0x41bf27
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0000467e8 sp=0xc0000467e0 pc=0x4770c1
created by runtime.createfing in goroutine 1
	/home/ec2-user/repos/go/src/runtime/mfinal.go:163 +0x3d

goroutine 19 [trace reader (blocked)]:
runtime.gopark(0xc000046f78?, 0xc0000b64e0?, 0x80?, 0x40?, 0x10?)
	/home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc000046f38 sp=0xc000046f18 pc=0x43d04e
runtime.ReadTrace()
	/home/ec2-user/repos/go/src/runtime/trace.go:445 +0x31 fp=0xc000046fa0 sp=0xc000046f38 pc=0x45d4f1
runtime/trace.Start.func1()
	/home/ec2-user/repos/go/src/runtime/trace/trace.go:130 +0x45 fp=0xc000046fe0 sp=0xc000046fa0 pc=0x4eeca5
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000046fe8 sp=0xc000046fe0 pc=0x4770c1
created by runtime/trace.Start in goroutine 1
	/home/ec2-user/repos/go/src/runtime/trace/trace.go:128 +0xdf

goroutine 20 [chan receive]:
runtime.gopark(0xc0000476e8?, 0x40fa25?, 0xa0?, 0x60?, 0x20?)
	/home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc000047680 sp=0xc000047660 pc=0x43d04e
runtime.chanrecv(0xc00010a000, 0x0, 0x1)
	/home/ec2-user/repos/go/src/runtime/chan.go:583 +0x3cd fp=0xc0000476f8 sp=0xc000047680 pc=0x408bed
runtime.chanrecv1(0xad3440?, 0x0?)
	/home/ec2-user/repos/go/src/runtime/chan.go:442 +0x12 fp=0xc000047720 sp=0xc0000476f8 pc=0x4087f2
runtime_test.TestDeferHeapAndStack(0x0?)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:898 +0x172 fp=0xc000047770 sp=0xc000047720 pc=0x75a292
testing.tRunner(0xc000083040, 0x857958)
	/home/ec2-user/repos/go/src/testing/testing.go:1595 +0xff fp=0xc0000477c0 sp=0xc000047770 pc=0x4ff53f
testing.(*T).Run.func1()
	/home/ec2-user/repos/go/src/testing/testing.go:1648 +0x25 fp=0xc0000477e0 sp=0xc0000477c0 pc=0x500505
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0000477e8 sp=0xc0000477e0 pc=0x4770c1
created by testing.(*T).Run in goroutine 1
	/home/ec2-user/repos/go/src/testing/testing.go:1648 +0x3ad

goroutine 5 [runnable]:
runtime_test.deferHeapAndStack(0x37)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:903 +0x229 fp=0xc0001bc0c0 sp=0xc0001bc0b8 pc=0x75a5c9
runtime_test.deferHeapAndStack(0x38)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bc1a0 sp=0xc0001bc0c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x39)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bc280 sp=0xc0001bc1a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3a)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bc360 sp=0xc0001bc280 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bc440 sp=0xc0001bc360 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bc520 sp=0xc0001bc440 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bc600 sp=0xc0001bc520 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bc6e0 sp=0xc0001bc600 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x3f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bc7c0 sp=0xc0001bc6e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x40)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bc8a0 sp=0xc0001bc7c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x41)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bc980 sp=0xc0001bc8a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x42)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bca60 sp=0xc0001bc980 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x43)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bcb40 sp=0xc0001bca60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x44)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bcc20 sp=0xc0001bcb40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x45)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bcd00 sp=0xc0001bcc20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x46)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bcde0 sp=0xc0001bcd00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x47)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bcec0 sp=0xc0001bcde0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x48)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bcfa0 sp=0xc0001bcec0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x49)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bd080 sp=0xc0001bcfa0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4a)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bd160 sp=0xc0001bd080 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bd240 sp=0xc0001bd160 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bd320 sp=0xc0001bd240 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bd400 sp=0xc0001bd320 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bd4e0 sp=0xc0001bd400 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x4f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bd5c0 sp=0xc0001bd4e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x50)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bd6a0 sp=0xc0001bd5c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x51)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bd780 sp=0xc0001bd6a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x52)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bd860 sp=0xc0001bd780 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x53)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bd940 sp=0xc0001bd860 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x54)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bda20 sp=0xc0001bd940 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x55)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bdb00 sp=0xc0001bda20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x56)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bdbe0 sp=0xc0001bdb00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x57)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bdcc0 sp=0xc0001bdbe0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x58)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bdda0 sp=0xc0001bdcc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x59)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bde80 sp=0xc0001bdda0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5a)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001bdf60 sp=0xc0001bde80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be040 sp=0xc0001bdf60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be120 sp=0xc0001be040 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be200 sp=0xc0001be120 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be2e0 sp=0xc0001be200 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be3c0 sp=0xc0001be2e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x60)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be4a0 sp=0xc0001be3c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x61)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be580 sp=0xc0001be4a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x62)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be660 sp=0xc0001be580 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x63)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be740 sp=0xc0001be660 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x64)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be820 sp=0xc0001be740 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x65)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be900 sp=0xc0001be820 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x66)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001be9e0 sp=0xc0001be900 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x67)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001beac0 sp=0xc0001be9e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x68)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001beba0 sp=0xc0001beac0 pc=0x75a4e9
...48 frames elided...
runtime_test.deferHeapAndStack(0x99)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c1680 sp=0xc0001c15a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9a)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c1760 sp=0xc0001c1680 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c1840 sp=0xc0001c1760 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c1920 sp=0xc0001c1840 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c1a00 sp=0xc0001c1920 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c1ae0 sp=0xc0001c1a00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c1bc0 sp=0xc0001c1ae0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c1ca0 sp=0xc0001c1bc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c1d80 sp=0xc0001c1ca0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c1e60 sp=0xc0001c1d80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c1f40 sp=0xc0001c1e60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2020 sp=0xc0001c1f40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2100 sp=0xc0001c2020 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c21e0 sp=0xc0001c2100 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c22c0 sp=0xc0001c21e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c23a0 sp=0xc0001c22c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa9)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2480 sp=0xc0001c23a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaa)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2560 sp=0xc0001c2480 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xab)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2640 sp=0xc0001c2560 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xac)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2720 sp=0xc0001c2640 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xad)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2800 sp=0xc0001c2720 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xae)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c28e0 sp=0xc0001c2800 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaf)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c29c0 sp=0xc0001c28e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2aa0 sp=0xc0001c29c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2b80 sp=0xc0001c2aa0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2c60 sp=0xc0001c2b80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2d40 sp=0xc0001c2c60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2e20 sp=0xc0001c2d40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2f00 sp=0xc0001c2e20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c2fe0 sp=0xc0001c2f00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c30c0 sp=0xc0001c2fe0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c31a0 sp=0xc0001c30c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb9)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3280 sp=0xc0001c31a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xba)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3360 sp=0xc0001c3280 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbb)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3440 sp=0xc0001c3360 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbc)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3520 sp=0xc0001c3440 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbd)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3600 sp=0xc0001c3520 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbe)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c36e0 sp=0xc0001c3600 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbf)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c37c0 sp=0xc0001c36e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c38a0 sp=0xc0001c37c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3980 sp=0xc0001c38a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3a60 sp=0xc0001c3980 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3b40 sp=0xc0001c3a60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3c20 sp=0xc0001c3b40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3d00 sp=0xc0001c3c20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3de0 sp=0xc0001c3d00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3ec0 sp=0xc0001c3de0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001c3fa0 sp=0xc0001c3ec0 pc=0x75a4e9
runtime_test.TestDeferHeapAndStack.func1()
	/home/ec2-user/repos/go/src/runtime/stack_test.go:890 +0x5a fp=0xc0001c3fe0 sp=0xc0001c3fa0 pc=0x75a35a
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0001c3fe8 sp=0xc0001c3fe0 pc=0x4770c1
created by runtime_test.TestDeferHeapAndStack in goroutine 20
	/home/ec2-user/repos/go/src/runtime/stack_test.go:888 +0xf3

goroutine 6 [runnable]:
runtime_test.deferHeapAndStack(0x5b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:903 +0x229 fp=0xc000176040 sp=0xc000176038 pc=0x75a5c9
runtime_test.deferHeapAndStack(0x5c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176120 sp=0xc000176040 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176200 sp=0xc000176120 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001762e0 sp=0xc000176200 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x5f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001763c0 sp=0xc0001762e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x60)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001764a0 sp=0xc0001763c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x61)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176580 sp=0xc0001764a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x62)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176660 sp=0xc000176580 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x63)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176740 sp=0xc000176660 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x64)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176820 sp=0xc000176740 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x65)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176900 sp=0xc000176820 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x66)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001769e0 sp=0xc000176900 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x67)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176ac0 sp=0xc0001769e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x68)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176ba0 sp=0xc000176ac0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x69)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176c80 sp=0xc000176ba0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x6a)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176d60 sp=0xc000176c80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x6b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176e40 sp=0xc000176d60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x6c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000176f20 sp=0xc000176e40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x6d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177000 sp=0xc000176f20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x6e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001770e0 sp=0xc000177000 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x6f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001771c0 sp=0xc0001770e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x70)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001772a0 sp=0xc0001771c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x71)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177380 sp=0xc0001772a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x72)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177460 sp=0xc000177380 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x73)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177540 sp=0xc000177460 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x74)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177620 sp=0xc000177540 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x75)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177700 sp=0xc000177620 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x76)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001777e0 sp=0xc000177700 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x77)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001778c0 sp=0xc0001777e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x78)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001779a0 sp=0xc0001778c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x79)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177a80 sp=0xc0001779a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x7a)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177b60 sp=0xc000177a80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x7b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177c40 sp=0xc000177b60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x7c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177d20 sp=0xc000177c40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x7d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177e00 sp=0xc000177d20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x7e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177ee0 sp=0xc000177e00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x7f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000177fc0 sp=0xc000177ee0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x80)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001780a0 sp=0xc000177fc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x81)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000178180 sp=0xc0001780a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x82)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000178260 sp=0xc000178180 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x83)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000178340 sp=0xc000178260 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x84)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000178420 sp=0xc000178340 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x85)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000178500 sp=0xc000178420 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x86)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001785e0 sp=0xc000178500 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x87)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001786c0 sp=0xc0001785e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x88)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0001787a0 sp=0xc0001786c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x89)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000178880 sp=0xc0001787a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8a)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000178960 sp=0xc000178880 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000178a40 sp=0xc000178960 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000178b20 sp=0xc000178a40 pc=0x75a4e9
...12 frames elided...
runtime_test.deferHeapAndStack(0x99)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000179680 sp=0xc0001795a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9a)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000179760 sp=0xc000179680 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000179840 sp=0xc000179760 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000179920 sp=0xc000179840 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000179a00 sp=0xc000179920 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000179ae0 sp=0xc000179a00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000179bc0 sp=0xc000179ae0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000179ca0 sp=0xc000179bc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000179d80 sp=0xc000179ca0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000179e60 sp=0xc000179d80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000179f40 sp=0xc000179e60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a020 sp=0xc000179f40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a100 sp=0xc00017a020 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a1e0 sp=0xc00017a100 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a2c0 sp=0xc00017a1e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a3a0 sp=0xc00017a2c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa9)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a480 sp=0xc00017a3a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaa)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a560 sp=0xc00017a480 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xab)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a640 sp=0xc00017a560 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xac)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a720 sp=0xc00017a640 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xad)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a800 sp=0xc00017a720 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xae)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a8e0 sp=0xc00017a800 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaf)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017a9c0 sp=0xc00017a8e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017aaa0 sp=0xc00017a9c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017ab80 sp=0xc00017aaa0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017ac60 sp=0xc00017ab80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017ad40 sp=0xc00017ac60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017ae20 sp=0xc00017ad40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017af00 sp=0xc00017ae20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017afe0 sp=0xc00017af00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017b0c0 sp=0xc00017afe0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017b1a0 sp=0xc00017b0c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb9)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017b280 sp=0xc00017b1a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xba)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017b360 sp=0xc00017b280 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbb)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017b440 sp=0xc00017b360 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbc)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017b520 sp=0xc00017b440 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbd)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017b600 sp=0xc00017b520 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbe)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017b6e0 sp=0xc00017b600 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbf)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017b7c0 sp=0xc00017b6e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017b8a0 sp=0xc00017b7c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017b980 sp=0xc00017b8a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017ba60 sp=0xc00017b980 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017bb40 sp=0xc00017ba60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017bc20 sp=0xc00017bb40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017bd00 sp=0xc00017bc20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017bde0 sp=0xc00017bd00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017bec0 sp=0xc00017bde0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc00017bfa0 sp=0xc00017bec0 pc=0x75a4e9
runtime_test.TestDeferHeapAndStack.func1()
	/home/ec2-user/repos/go/src/runtime/stack_test.go:890 +0x5a fp=0xc00017bfe0 sp=0xc00017bfa0 pc=0x75a35a
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00017bfe8 sp=0xc00017bfe0 pc=0x4770c1
created by runtime_test.TestDeferHeapAndStack in goroutine 20
	/home/ec2-user/repos/go/src/runtime/stack_test.go:888 +0xf3

goroutine 8 [runnable]:
runtime.newobject(0x7a6640?)
	/home/ec2-user/repos/go/src/runtime/malloc.go:1260 +0x35 fp=0xc0004625e0 sp=0xc0004625d8 pc=0x40fdb5
runtime_test.deferHeapAndStack(0x87)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:924 +0x165 fp=0xc0004626c0 sp=0xc0004625e0 pc=0x75a505
runtime_test.deferHeapAndStack(0x88)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004627a0 sp=0xc0004626c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x89)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000462880 sp=0xc0004627a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8a)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000462960 sp=0xc000462880 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000462a40 sp=0xc000462960 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000462b20 sp=0xc000462a40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000462c00 sp=0xc000462b20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000462ce0 sp=0xc000462c00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x8f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000462dc0 sp=0xc000462ce0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x90)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000462ea0 sp=0xc000462dc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x91)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000462f80 sp=0xc000462ea0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x92)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463060 sp=0xc000462f80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x93)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463140 sp=0xc000463060 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x94)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463220 sp=0xc000463140 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x95)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463300 sp=0xc000463220 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x96)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004633e0 sp=0xc000463300 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x97)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004634c0 sp=0xc0004633e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x98)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004635a0 sp=0xc0004634c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x99)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463680 sp=0xc0004635a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9a)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463760 sp=0xc000463680 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9b)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463840 sp=0xc000463760 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9c)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463920 sp=0xc000463840 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9d)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463a00 sp=0xc000463920 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9e)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463ae0 sp=0xc000463a00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0x9f)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463bc0 sp=0xc000463ae0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463ca0 sp=0xc000463bc0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463d80 sp=0xc000463ca0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463e60 sp=0xc000463d80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000463f40 sp=0xc000463e60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464020 sp=0xc000463f40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464100 sp=0xc000464020 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004641e0 sp=0xc000464100 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004642c0 sp=0xc0004641e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004643a0 sp=0xc0004642c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xa9)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464480 sp=0xc0004643a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaa)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464560 sp=0xc000464480 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xab)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464640 sp=0xc000464560 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xac)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464720 sp=0xc000464640 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xad)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464800 sp=0xc000464720 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xae)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004648e0 sp=0xc000464800 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xaf)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004649c0 sp=0xc0004648e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464aa0 sp=0xc0004649c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464b80 sp=0xc000464aa0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464c60 sp=0xc000464b80 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464d40 sp=0xc000464c60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464e20 sp=0xc000464d40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464f00 sp=0xc000464e20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000464fe0 sp=0xc000464f00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004650c0 sp=0xc000464fe0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004651a0 sp=0xc0004650c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xb9)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465280 sp=0xc0004651a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xba)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465360 sp=0xc000465280 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbb)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465440 sp=0xc000465360 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbc)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465520 sp=0xc000465440 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbd)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465600 sp=0xc000465520 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbe)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004656e0 sp=0xc000465600 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xbf)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004657c0 sp=0xc0004656e0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc0)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc0004658a0 sp=0xc0004657c0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc1)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465980 sp=0xc0004658a0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc2)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465a60 sp=0xc000465980 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc3)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465b40 sp=0xc000465a60 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc4)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465c20 sp=0xc000465b40 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc5)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465d00 sp=0xc000465c20 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc6)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465de0 sp=0xc000465d00 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc7)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465ec0 sp=0xc000465de0 pc=0x75a4e9
runtime_test.deferHeapAndStack(0xc8)
	/home/ec2-user/repos/go/src/runtime/stack_test.go:923 +0x149 fp=0xc000465fa0 sp=0xc000465ec0 pc=0x75a4e9
runtime_test.TestDeferHeapAndStack.func1()
	/home/ec2-user/repos/go/src/runtime/stack_test.go:890 +0x5a fp=0xc000465fe0 sp=0xc000465fa0 pc=0x75a35a
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000465fe8 sp=0xc000465fe0 pc=0x4770c1
created by runtime_test.TestDeferHeapAndStack in goroutine 20
	/home/ec2-user/repos/go/src/runtime/stack_test.go:888 +0xf3

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0xc000380000?)
	/home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00004cf50 sp=0xc00004cf30 pc=0x43d04e
runtime.gcBgMarkWorker()
	/home/ec2-user/repos/go/src/runtime/mgc.go:1286 +0xe5 fp=0xc00004cfe0 sp=0xc00004cf50 pc=0x41eb45
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00004cfe8 sp=0xc00004cfe0 pc=0x4770c1
created by runtime.gcBgMarkStartWorkers in goroutine 7
	/home/ec2-user/repos/go/src/runtime/mgc.go:1210 +0x1c

goroutine 50 [GC worker (idle)]:
runtime.gopark(0x856450?, 0xc00051e020?, 0x1a?, 0x14?, 0x0?)
	/home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc00004bf50 sp=0xc00004bf30 pc=0x43d04e
runtime.gcBgMarkWorker()
	/home/ec2-user/repos/go/src/runtime/mgc.go:1286 +0xe5 fp=0xc00004bfe0 sp=0xc00004bf50 pc=0x41eb45
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00004bfe8 sp=0xc00004bfe0 pc=0x4770c1
created by runtime.gcBgMarkStartWorkers in goroutine 7
	/home/ec2-user/repos/go/src/runtime/mgc.go:1210 +0x1c

goroutine 51 [semacquire]:
runtime.gopark(0xc00051c360?, 0x4822?, 0xc0?, 0xc3?, 0x44e2ef?)
	/home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc000526680 sp=0xc000526660 pc=0x43d04e
runtime.goparkunlock(...)
	/home/ec2-user/repos/go/src/runtime/proc.go:392
runtime.semacquire1(0xaf54bc, 0x0?, 0x0, 0x0, 0xd3?)
	/home/ec2-user/repos/go/src/runtime/sema.go:160 +0x20f fp=0xc0005266f0 sp=0xc000526680 pc=0x44e0cf
runtime.semacquire(...)
	/home/ec2-user/repos/go/src/runtime/sema.go:111
runtime.gcMarkDone()
	/home/ec2-user/repos/go/src/runtime/mgc.go:786 +0x2a fp=0xc000526750 sp=0xc0005266f0 pc=0x41d90a
runtime.gcBgMarkWorker()
	/home/ec2-user/repos/go/src/runtime/mgc.go:1418 +0x345 fp=0xc0005267e0 sp=0xc000526750 pc=0x41eda5
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0005267e8 sp=0xc0005267e0 pc=0x4770c1
created by runtime.gcBgMarkStartWorkers in goroutine 7
	/home/ec2-user/repos/go/src/runtime/mgc.go:1210 +0x1c

goroutine 52 [semacquire]:
runtime.gopark(0x8a6bc?, 0x0?, 0x0?, 0x0?, 0x3?)
	/home/ec2-user/repos/go/src/runtime/proc.go:386 +0xce fp=0xc000526e80 sp=0xc000526e60 pc=0x43d04e
runtime.goparkunlock(...)
	/home/ec2-user/repos/go/src/runtime/proc.go:392
runtime.semacquire1(0xaf54bc, 0x98?, 0x0, 0x0, 0x30?)
	/home/ec2-user/repos/go/src/runtime/sema.go:160 +0x20f fp=0xc000526ef0 sp=0xc000526e80 pc=0x44e0cf
runtime.semacquire(...)
	/home/ec2-user/repos/go/src/runtime/sema.go:111
runtime.gcMarkDone()
	/home/ec2-user/repos/go/src/runtime/mgc.go:786 +0x2a fp=0xc000526f50 sp=0xc000526ef0 pc=0x41d90a
runtime.gcBgMarkWorker()
	/home/ec2-user/repos/go/src/runtime/mgc.go:1418 +0x345 fp=0xc000526fe0 sp=0xc000526f50 pc=0x41eda5
runtime.goexit()
	/home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000526fe8 sp=0xc000526fe0 pc=0x4770c1
created by runtime.gcBgMarkStartWorkers in goroutine 7
	/home/ec2-user/repos/go/src/runtime/mgc.go:1210 +0x1c

rax    0x6
rbx    0x7fafc0092018
rcx    0x80
rdx    0x2
rdi    0x80
rsi    0x1
rbp    0x7fafbb5fdcb8
rsp    0x7fafbb5fdc80
r8     0xc000007d40
r9     0x7fafc0092028
r10    0x7e
r11    0xc000080800
r12    0x7e6c
r13    0x0
r14    0xc0000821a0
r15    0x1
rip    0x45ead7
rflags 0x10202
cs     0x33
fs     0x0
gs     0x0
exit status 2
FAIL	runtime	0.043s

My first thought is to double check the various stack-switching routines (systemstack, mcall, morestack) to see if they're writing stuff where the frame pointer should be, or not setting a new frame pointer up properly when taking over the stack.

@mknyszek mknyszek changed the title runtime: fpTracebackPCs crash on linux/amd64 while building kubelet runtime: fpTracebackPCs crash on linux/amd64 in various places Apr 19, 2023
@prattmic
Copy link
Member

Given we have a reliable reproducer, I think we should consider disabling frame pointer unwinding at HEAD until we have a fix. (I'd say flip the default value of debug.tracefpunwindoff rather than revert the entire CL).

@felixge
Copy link
Contributor

felixge commented Apr 19, 2023

@prattmic I'm planning to look into this as soon as I'm back from vacation on Monday. That being said, I'm okay with disabling debug.tracefpunwindoff until then.

One thing I'm curious about is if this issue is also caught by debugCheckBP? If yes, this would strengthen my belief that we should attempt turning this on by default (and fixing whatever else it might reveal) in order to have reliable frame pointers.

@nsrip-dd 🙇 thanks for finding a reproducer!

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/486382 mentions this issue: runtime: turn frame pointer unwinding for tracing off by default

@felixge
Copy link
Contributor

felixge commented Apr 20, 2023

One thing I'm curious about is if this issue is also caught by debugCheckBP?

Just tried it with debugCheckBP = true, and it seems like it doesn't crash:

GODEBUG=tracefpunwindoff=1 ../bin/go test -trace /dev/null -run TestDeferHeapAndStack runtime
ok  	runtime	2.413s

gopherbot pushed a commit that referenced this issue Apr 20, 2023
Frame pointer unwinding during execution tracing sometimes crashes.
Until this is diagnosed and fixed, it should be turned off by default.

Updates #59692

Change-Id: I0f2ca24b6d48435b0acfd3da8e4f25b9cfa4ec19
Reviewed-on: https://go-review.googlesource.com/c/go/+/486382
Reviewed-by: Felix Geisendörfer <felix.geisendoerfer@datadoghq.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Nick Ripley <nick.ripley@datadoghq.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Michael Pratt <mpratt@google.com>
@nsrip-dd
Copy link
Contributor

I've dug into this a little bit. I don't have any conclusive answer, but I think what's happening involves an incorrect value ending up in the RBP register at some point during a call to gcAssistAlloc. Below are some loosely organized notes from my debugging, hopefully pointing us in a good direction. Note that I don't always get exactly the same crash, but when things do go bad it seems to consistently be around gcAssistAlloc.

I built the test binary and ran it under gdb, just doing TestDeferHeapAndStack with tracing enabled. I got a crash here:

(gdb) bt
#0  runtime.traceStackID (mp=<optimized out>, pcBuf=[]uintptr = {...}, skip=<optimized out>, ~r0=<optimized out>) at /home/ec2-user/repos/go/src/runtime/trace.go:916
#1  0x000000000045e0a5 in runtime.traceEventLocked (extraBytes=<optimized out>, mp=0xc00004d800, pid=<optimized out>, bufp=0xc0000351d0, ev=18 '\022', stackID=0, skip=1,
    args=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /home/ec2-user/repos/go/src/runtime/trace.go:759
#2  0x000000000045dda9 in runtime.traceEvent (ev=18 '\022', skip=1, args=[]uint64) at /home/ec2-user/repos/go/src/runtime/trace.go:691
#3  0x0000000000443b45 in runtime.traceGoPreempt () at /home/ec2-user/repos/go/src/runtime/trace.go:1534
#4  runtime.gopreempt_m (gp=0xc0000824e0) at /home/ec2-user/repos/go/src/runtime/proc.go:3550
#5  0x000000000045514b in runtime.newstack () at /home/ec2-user/repos/go/src/runtime/stack.go:1090
#6  0x000000000047526f in runtime.morestack () at /home/ec2-user/repos/go/src/runtime/asm_amd64.s:593
#7  0x0000000000000000 in ?? ()

The crash is happening during frame pointer unwinding, at this instruction (where we're looking for the PC relative to the current frame pointer):

  45ead7:       mov    0x8(%rsi),%r11 <==== crash here
  45eadb:       mov    %r11,(%r9,%rax,8)
  45eadf:       mov    (%rsi),%rsi
  45eae2:       inc    %rax

Now, we can see right away that curgp.sched.bp (where we're going to start unwinding) does not look right:

(gdb) p/x curgp.sched.bp
$23 = 0xc0000bd858
(gdb) p/x curgp.stack
$25 = {lo = 0xc0003ee000, hi = 0xc0003f6000}

(This makes me wonder why debugCheckBP = true isn't complaining -- it even has a check of curgp.sched.bp, which presumably should fail here?)

Here's where morestack was called from:

(gdb) p/a curgp.sched.pc
$21 = 0x41dbd0 <runtime.gcMarkDone+752>

So morestack is being called from runtime.gcMarkDone. This happens at the beginning of gcMarkDone, before the frame is set up. We know that morestack sets up curgp.sched.sp to point the the caller's frame, just above where the return address is pushed (i.e. the stack pointer as it would have been immediately prior to the CALL instruction, which pushes the return address).

Since we're at the begging of runtime.gcMarkDone, we know that the stack pointer should thus be pointing to the return address that was pushed by runtime.gcMarkDone's caller:

(gdb) x/a curgp.sched.sp
0xc0003f3800:   0x4217e7 <runtime.gcAssistAlloc+487>

Now, we can look at the disassembly of runtime.gcAssistAlloc to see how big of a frame it uses:

0000000000421600 <runtime.gcAssistAlloc>:
  421600:       cmp    0x10(%r14),%rsp
  421604:       jbe    4218ab <runtime.gcAssistAlloc+0x2ab>
  42160a:       push   %rbp
  42160b:       mov    %rsp,%rbp
  42160e:       sub    $0x50,%rsp

So it uses a 0x50 byte frame. Note that this is after pushing RBP to the stack. Let's see what was saved there. It should be 0x50 bytes off of curgp.sched.sp, plus an additional 8 bytes because we're pointing to where the return address within gcAssistAlloc is saved before calling gcMarkDone.

(gdb) p/x curgp.sched.sp+0x50+8
$22 = 0xc0003f3858

Let's see what the return address of gcAssistAlloc is, which should be an additional 8 bytes above the saved frame pointer:

(gdb) x/a curgp.sched.sp+0x50+8+8
0xc0003f3860:   0x40fcb4 <runtime.deductAssistCredit+84>

This looks right! Now, let's check the saved frame pointer:

(gdb) x/g curgp.sched.sp+0x50+8
0xc0003f3858:   0x000000c0003f3880

This is promising. The address 0xc0003f3880 is within our stack bounds. Let's see if there's a return address saved right above this address, assuming it's a valid frame pointer:

(gdb) x/a 0x000000c0003f3880+8
0xc0003f3888:   0x40f4e9 <runtime.mallocgc+201>

This checks out too!

40f4df:       mov    %rax,0x68(%rsp)
40f4e4:       callq  40fc60 <runtime.deductAssistCredit>
40f4e9:       mov    0x30(%r14),%rcx

So, despite curgp.sched.bp being wrong, it looks like the frame pointers saved on the stack are right. So, why is curgp.sched.bp wrong?

Let's consider how curgp.sched.bp is set. Here's where it's set up by morestack:

TEXT runtime·morestack(SB),NOSPLIT|NOFRAME,$0-0
	... elided ...

	// Set g->sched to context in f.
	MOVQ	0(SP), AX // f's PC
	MOVQ	AX, (g_sched+gobuf_pc)(SI)
	LEAQ	8(SP), AX // f's SP
	MOVQ	AX, (g_sched+gobuf_sp)(SI)
	MOVQ	BP, (g_sched+gobuf_bp)(SI)
	MOVQ	DX, (g_sched+gobuf_ctxt)(SI)

So, it's set to be the current value of RBP when morestack is called. Note that morestack is marked NOFRAME, so it doesn't create a new frame pointer. It's jumped to from morestack_noctxt:

// morestack but not preserving ctxt.
TEXT runtime·morestack_noctxt(SB),NOSPLIT,$0
	MOVL	$0, DX
	JMP	runtime·morestack(SB)

0000000000475280 <runtime.morestack_noctxt.abi0>:
  475280:       mov    $0x0,%edx
  475285:       jmpq   4751e0 <runtime.morestack.abi0>

So, this means we're going to use the value of BP as it was immediately prior to gcMarkDone being called within gcAssitAlloc. At some point between when gcAssistAlloc was called, and when we reached the gcMarkDone call, something invalid was put in BP. I think based on the analysis above, gcAssistAlloc would have set up the correct frame pointer. My hunch is that something in between overwrote it incorrectly.


One idea I'm trying out: If my hunch is that RBP is invalid at some point after gcAssistAlloc starts, perhaps let's stop around where we think it might be invalid and see what its value is.

I set a breakpoint in gcAssistAlloc just before gcMarkDone is called. Then I had gdb display RBP and RSP. I think we should always have RBP >= RSP during normal code (so potentially not for a few instructions during stack switching or something). They should probably also be close-ish together, since in theory RBP just points to the top of the frame and RSP points to the bottom.

A typical case looks like this:

Thread 5 "runtime.test" hit Breakpoint 6, runtime.gcAssistAlloc (gp=0xc000185380) at /home/ec2-user/repos/go/src/runtime/mgcmark.go:483
483                     gcMarkDone()
2: $rbp = (void *) 0xc000251eb8
3: $rsp = (void *) 0xc000251e68

As noted above, these values differ by 0x50, the size of the frame that gcAssitAlloc sets up. So this seems to be right.

Here's one case where that wasn't true:

Thread 8 "runtime.test" hit Breakpoint 6, runtime.gcAssistAlloc (gp=0xc000185380) at /home/ec2-user/repos/go/src/runtime/mgcmark.go:483
483                     gcMarkDone()
2: $rbp = (void *) 0xc00025a658
3: $rsp = (void *) 0xc0003be608

Notice how different the two values are.

Here's the stack trace at this point:

(gdb) bt
#0  runtime.gcAssistAlloc (gp=0xc000185380) at /home/ec2-user/repos/go/src/runtime/mgcmark.go:483
#1  0x000000000040fcb4 in runtime.deductAssistCredit (size=<optimized out>, ~r0=<optimized out>) at /home/ec2-user/repos/go/src/runtime/malloc.go:1224
#2  0x000000000040f4e9 in runtime.mallocgc (size=8, typ=0x7a4ac0, needzero=true, ~r0=<optimized out>) at /home/ec2-user/repos/go/src/runtime/malloc.go:939
#3  0x000000000040fda5 in runtime.newobject (typ=<optimized out>, ~r0=<optimized out>) at /home/ec2-user/repos/go/src/runtime/malloc.go:1261
#4  0x000000000075a3dd in runtime_test.deferHeapAndStack (n=173, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:903
#5  0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=174, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#6  0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=175, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#7  0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=176, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#8  0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=177, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#9  0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=178, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#10 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=179, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#11 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=180, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#12 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=181, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#13 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=182, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#14 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=183, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#15 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=184, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#16 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=185, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#17 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=186, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#18 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=187, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#19 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=188, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#20 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=189, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#21 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=190, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#22 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=191, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#23 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=192, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#24 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=193, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#25 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=194, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#26 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=195, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#27 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=196, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#28 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=197, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#29 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=198, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#30 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=199, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#31 0x000000000075a4e9 in runtime_test.deferHeapAndStack (n=200, r=<optimized out>) at /home/ec2-user/repos/go/src/runtime/stack_test.go:923
#32 0x000000000075a35a in runtime_test.TestDeferHeapAndStack.func1 () at /home/ec2-user/repos/go/src/runtime/stack_test.go:890
#33 0x00000000004770c1 in runtime.goexit () at /home/ec2-user/repos/go/src/runtime/asm_amd64.s:1622
#34 0x0000000000000000 in ?? ()

This is where I'm stumped for now. I don't yet have a machine that can run rr, but I think that would be helpful to get to a point like this where the frame pointer seems to be invalid and go backwards to the places where it is updated.

@mknyszek
Copy link
Contributor Author

mknyszek commented Apr 21, 2023

@nsrip-dd Nice deep dive!

After reading your analysis, I had a suspicion that maybe the gcAssistAlloc goroutine is getting its stack shrunk (and it's getting scanned while it's in gcAssistAlloc1). Sure enough, I could reproduce with stackDebug=1 and the dying goroutine has its stack very recently shrunk. I'll check the frame pointer before and after.

EDIT: To reproduce I had to shrink GOMAXPROCS on my machine to 8. If it was higher than that it wouldn't reproduce. Maybe it lowered the chance that assists took care of stack shrinking and copying, since there were more idle-priority GC workers?

@mknyszek
Copy link
Contributor Author

mknyszek commented Apr 21, 2023

I think I've confirmed that there's consistently a stack shrink on the relevant goroutine specifically while it's in gcAssistAlloc1 when the failure happens. Unfortunately gp.sched.bp appears to be correct going in and coming out, so that's not obviously the problem.

stackDebug=1 (augmented)
...
entering gcAssistAlloc1 23
entering gcAssistAlloc1 21
shrinking stack 65536->32768
stackalloc 32768
  allocated 0xc0004e8000
copystack gp=0xc000102ea0 (23) [0xc00013a000 0xc0001493f0 0xc00014a000] -> [0xc0004e8000 0xc0004ef3f0 0xc0004f0000]/32768
  bp before 23 0xc0001493f0
runtime: newstack sp=0xc0000ca710 stack=[0xc0000bc000, 0xc0000cc000]
	morebuf={pc:0x75ba05 sp:0xc0000ca720 lr:0x0}
	sched={pc:0x40ff15 sp:0xc0000ca718 lr:0x0 ctxt:0x0}
shrinking stack 65536->32768
stackalloc 32768
  allocated 0xc00053c000
copystack gp=0xc000102d00 (22) [0xc0000bc000 0xc0000ca718 0xc0000cc000] -> [0xc00053c000 0xc000542718 0xc000544000]/32768
  bp before 22 0xc0000ca7f0
exiting gcAssistAlloc1 21
stackfree 0xc0000bc000 65536
  bp after 22 0xc0005427f0
entering gcAssistAlloc1 20
stackfree 0xc00013a000 65536
exiting gcAssistAlloc1 20
entering gcAssistAlloc1 21
  bp after 23 0xc0004ef3f0
runtime: newstack sp=0xc0004ef3f0 stack=[0xc0004e8000, 0xc0004f0000]
	morebuf={pc:0x421839 sp:0xc0004ef400 lr:0x0}
	sched={pc:0x43cb15 sp:0xc0004ef3f8 lr:0x0 ctxt:0x0}
runtime: newstack sp=0xc000543970 stack=[0xc00053c000, 0xc000544000]
	morebuf={pc:0x75ba25 sp:0xc000543980 lr:0x0}
	sched={pc:0x439d54 sp:0xc000543978 lr:0x0 ctxt:0x0}
exiting gcAssistAlloc1 21
runtime: newstack sp=0xc0000a6290 stack=[0xc00009c000, 0xc0000ac000]
	morebuf={pc:0x42192f sp:0xc0000a62a0 lr:0x0}
	sched={pc:0x45f188 sp:0xc0000a6298 lr:0x0 ctxt:0x0}
runtime: newstack sp=0xc00053c3a0 stack=[0xc00053c000, 0xc000544000]
	morebuf={pc:0x40fb3f sp:0xc00053c3b0 lr:0x0}
	sched={pc:0x417fb6 sp:0xc00053c3a8 lr:0x0 ctxt:0x0}
stackalloc 65536
  allocated 0xc00013a000
copystack gp=0xc000102d00 (22) [0xc00053c000 0xc00053c3a8 0xc000544000] -> [0xc00013a000 0xc0001423a8 0xc00014a000]/65536
  bp before 22 0xc00053c408
runtime: newstack sp=0xc00041c9f0 stack=[0xc000416000, 0xc000426000]
	morebuf={pc:0x75ba25 sp:0xc00041ca00 lr:0x0}
	sched={pc:0x439d54 sp:0xc00041c9f8 lr:0x0 ctxt:0x0}
runtime: newstack sp=0xc0000aab70 stack=[0xc00009c000, 0xc0000ac000]
	morebuf={pc:0x75ba25 sp:0xc0000aab80 lr:0x0}
	sched={pc:0x439d54 sp:0xc0000aab78 lr:0x0 ctxt:0x0}
runtime: newstack sp=0xc0001b36e0 stack=[0xc0001b3000, 0xc0001b3800]
	morebuf={pc:0x41dac6 sp:0xc0001b36f0 lr:0x0}
	sched={pc:0x44f26e sp:0xc0001b36e8 lr:0x0 ctxt:0x0}
runtime: newstack sp=0xc0000a8170 stack=[0xc00009c000, 0xc0000ac000]
	morebuf={pc:0x75ba68 sp:0xc0000a8180 lr:0x0}
	sched={pc:0x40ff15 sp:0xc0000a8178 lr:0x0 ctxt:0x0}
exiting gcAssistAlloc1 23
runtime: newstack sp=0xc0004ef3f0 stack=[0xc0004e8000, 0xc0004f0000]
	morebuf={pc:0x42192f sp:0xc0004ef400 lr:0x0}
	sched={pc:0x45f188 sp:0xc0004ef3f8 lr:0x0 ctxt:0x0}
runtime: newstack sp=0xc0000a5bd0 stack=[0xc00009c000, 0xc0000ac000]
	morebuf={pc:0x75ba05 sp:0xc0000a5be0 lr:0x0}
	sched={pc:0x40ff15 sp:0xc0000a5bd8 lr:0x0 ctxt:0x0}
stackfree 0xc00053c000 32768
  bp after 22 0xc000142408
stack grow done
runtime: newstack sp=0xc000142430 stack=[0xc00013a000, 0xc00014a000]
	morebuf={pc:0x75ba9e sp:0xc000142440 lr:0x0}
	sched={pc:0x439672 sp:0xc000142438 lr:0x0 ctxt:0x0}
runtime: newstack sp=0xc0004203b0 stack=[0xc000416000, 0xc000426000]
	morebuf={pc:0x75ba05 sp:0xc0004203c0 lr:0x0}
	sched={pc:0x40ff15 sp:0xc0004203b8 lr:0x0 ctxt:0x0}
SIGSEGV: segmentation violation
PC=0x45fe77 m=10 sigcode=1

goroutine 0 [idle]:
runtime.fpTracebackPCs(...)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:944
runtime.traceStackID(0x6c070ac4eaf43?, {0x7f85684d4018, 0x2?, 0x80}, 0x3?)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:916 +0x217 fp=0x7f8559ffab80 sp=0x7f8559ffab38 pc=0x45fe77
runtime.traceEventLocked(0xb31448?, 0x100000081?, 0x0?, 0xc0000485d0, 0x12, 0x0, 0x1, {0x0, 0x0, 0x0})
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:759 +0x285 fp=0x7f8559ffabf8 sp=0x7f8559ffab80 pc=0x45f445
runtime.traceEvent(0x65?, 0x1, {0x0, 0x0, 0x0})
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:691 +0xa9 fp=0x7f8559ffac60 sp=0x7f8559ffabf8 pc=0x45f149
runtime.traceGoPreempt(...)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:1534
runtime.gopreempt_m(0x82d691?)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/proc.go:3550 +0x45 fp=0x7f8559ffaca0 sp=0x7f8559ffac60 pc=0x4449c5
runtime.newstack()
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/stack.go:1095 +0x3d5 fp=0x7f8559ffae50 sp=0x7f8559ffaca0 pc=0x456375
runtime.morestack()
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/asm_amd64.s:593 +0x8f fp=0x7f8559ffae58 sp=0x7f8559ffae50 pc=0x47694f

goroutine 23 [running]:
runtime.traceEvent(0x2c?, 0xffffffffffffffff, {0x0, 0x0, 0x0})
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:668 +0xe8 fp=0xc0004ef400 sp=0xc0004ef3f8 pc=0x45f188
runtime.traceGCMarkAssistDone(...)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:1496
runtime.gcAssistAlloc(0xc000102ea0)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/mgcmark.go:518 +0x2ef fp=0xc0004ef468 sp=0xc0004ef400 pc=0x42192f
runtime.deductAssistCredit(0x0?)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/malloc.go:1287 +0x54 fp=0xc0004ef490 sp=0xc0004ef468 pc=0x40fe14
runtime.mallocgc(0x400, 0x7a8960, 0x1)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/malloc.go:1002 +0xc9 fp=0xc0004ef4f8 sp=0xc0004ef490 pc=0x40f649
runtime.newobject(0xbc?)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/malloc.go:1324 +0x25 fp=0xc0004ef520 sp=0xc0004ef4f8 pc=0x40ff05
runtime_test.deferHeapAndStack(0xbd)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/stack_test.go:924 +0x165 fp=0xc0004ef600 sp=0xc0004ef520 pc=0x75ba05
	...

@mknyszek
Copy link
Contributor Author

Hm. Perhaps that's a red herring? If I put fpTracebackPCs(unsafe.Pointer(gp.sched.bp), 0, pcs[:]) just after returning from gcAssistAlloc1 (back on the regular stack) it doesn't crash there.

The few most recent crashes I've seen seem to be in an attempt to trace a preemption at entry to traceEvent called from traceGCMarkAssistDone.

@mknyszek
Copy link
Contributor Author

Well, OK. This is interesting. I caught the bad BP just before the traceback happens, and I can confirm the bad BP corresponds to the old stack, before the shrink. So the shrink matters.

stackDebug=1 (augmented)
copystack gp=0xc0001036c0 (23) [0xc0003a4000 0xc0003b3cb0 0xc0003b4000] -> [0xc00063e000 0xc000645cb0 0xc000646000]/32768
  bp before 23 0xc0003b3cb0
entering gcAssistAlloc1 25
entering gcAssistAlloc1 24
exiting gcAssistAlloc1 24
exiting gcAssistAlloc1 25
runtime: newstack sp=0xc0000d86b0 stack=[0xc0000cc000, 0xc0000dc000]
	morebuf={pc:0x75bb05 sp:0xc0000d86c0 lr:0x0}
	sched={pc:0x40ff15 sp:0xc0000d86b8 lr:0x0 ctxt:0x0}
shrinking stack 65536->32768
stackalloc 32768
  allocated 0xc0003f2000
copystack gp=0xc000103ba0 (26) [0xc0000cc000 0xc0000d86b8 0xc0000dc000] -> [0xc0003f2000 0xc0003f66b8 0xc0003fa000]/32768
  bp before 26 0xc0000d8790
stackfree 0xc0000cc000 65536
  bp after 26 0xc0003f6790
entering gcAssistAlloc1 24
exiting gcAssistAlloc1 24
runtime: newstack sp=0xc0003e23a0 stack=[0xc0003e2000, 0xc0003ea000]
	morebuf={pc:0x40fb3f sp:0xc0003e23b0 lr:0x0}
	sched={pc:0x417fb6 sp:0xc0003e23a8 lr:0x0 ctxt:0x0}
stackalloc 65536
  allocated 0xc0000cc000
copystack gp=0xc000103a00 (25) [0xc0003e2000 0xc0003e23a8 0xc0003ea000] -> [0xc0000cc000 0xc0000d43a8 0xc0000dc000]/65536
  bp before 25 0xc0003e2408
stackfree 0xc0003a4000 65536
runtime: newstack sp=0xc0003f23a0 stack=[0xc0003f2000, 0xc0003fa000]
	morebuf={pc:0x40fb3f sp:0xc0003f23b0 lr:0x0}
	sched={pc:0x417fb6 sp:0xc0003f23a8 lr:0x0 ctxt:0x0}
stackalloc 65536
  allocated 0xc0003a4000
copystack gp=0xc000103ba0 (26) [0xc0003f2000 0xc0003f23a8 0xc0003fa000] -> [0xc0003a4000 0xc0003ac3a8 0xc0003b4000]/65536
  bp before 26 0xc0003f2408
stackfree 0xc0003e2000 32768
  bp after 25 0xc0000d4408
stack grow done
  bp after 23 0xc000645cb0
runtime: newstack sp=0xc000645cb0 stack=[0xc00063e000, 0xc000646000]
	morebuf={pc:0x421839 sp:0xc000645cc0 lr:0x0}
	sched={pc:0x43cb55 sp:0xc000645cb8 lr:0x0 ctxt:0x0}
runtime: newstack sp=0xc0000a6030 stack=[0xc00009c000, 0xc0000ac000]
	morebuf={pc:0x75bb25 sp:0xc0000a6040 lr:0x0}
	sched={pc:0x439d94 sp:0xc0000a6038 lr:0x0 ctxt:0x0}
tracing 24 0xc0000a6110
tracing done 24
tracing 23 0xc0003b3d18
runtime: newstack sp=0xc0000a9ad0 stack=[0xc00009c000, 0xc0000ac000]
	morebuf={pc:0x75bb25 sp:0xc0000a9ae0 lr:0x0}
	sched={pc:0x439d94 sp:0xc0000a9ad8 lr:0x0 ctxt:0x0}
tracing 24 0xc0000a9bb0
tracing done 24
runtime: newstack sp=0xc0000d4890 stack=[0xc0000cc000, 0xc0000dc000]
	morebuf={pc:0x75bb05 sp:0xc0000d48a0 lr:0x0}
	sched={pc:0x40ff15 sp:0xc0000d4898 lr:0x0 ctxt:0x0}
tracing 25 0xc0000d4970
tracing done 25
stackfree 0xc0003f2000 32768
  bp after 26 0xc0003ac408
stack grow done
runtime: newstack sp=0xc0003ac430 stack=[0xc0003a4000, 0xc0003b4000]
	morebuf={pc:0x75bb9e sp:0xc0003ac440 lr:0x0}
	sched={pc:0x4396b2 sp:0xc0003ac438 lr:0x0 ctxt:0x0}
tracing 26 0xc0003ac510
tracing done 26
SIGSEGV: segmentation violation
PC=0x45ff23 m=4 sigcode=1

goroutine 0 [idle]:
runtime.fpTracebackPCs(...)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:946
runtime.traceStackID(0x6c45ea277195d?, {0x7f8a0803b018, 0x80, 0x80}, 0x3?)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:917 +0x283 fp=0x7f8a0a27db80 sp=0x7f8a0a27db20 pc=0x45ff23
runtime.traceEventLocked(0xb31448?, 0x100000081?, 0x0?, 0xc000043bd0, 0x12, 0x0, 0x1, {0x0, 0x0, 0x0})
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:759 +0x285 fp=0x7f8a0a27dbf8 sp=0x7f8a0a27db80 pc=0x45f485
runtime.traceEvent(0x65?, 0x1, {0x0, 0x0, 0x0})
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:691 +0xa9 fp=0x7f8a0a27dc60 sp=0x7f8a0a27dbf8 pc=0x45f189
runtime.traceGoPreempt(...)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/trace.go:1536
runtime.gopreempt_m(0x82d691?)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/proc.go:3550 +0x45 fp=0x7f8a0a27dca0 sp=0x7f8a0a27dc60 pc=0x444a05
runtime.newstack()
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/stack.go:1095 +0x3d5 fp=0x7f8a0a27de50 sp=0x7f8a0a27dca0 pc=0x4563b5
runtime.morestack()
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/asm_amd64.s:593 +0x8f fp=0x7f8a0a27de58 sp=0x7f8a0a27de50 pc=0x476a4f

goroutine 23 [running]:
runtime.printlock()
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/print.go:65 +0x55 fp=0xc000645cc0 sp=0xc000645cb8 pc=0x43cb55
runtime.gcAssistAlloc(0xc0001036c0)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/mgcmark.go:482 +0x1f9 fp=0xc000645d28 sp=0xc000645cc0 pc=0x421839
runtime.deductAssistCredit(0xc0003b3d98?)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/malloc.go:1287 +0x54 fp=0xc000645d50 sp=0xc000645d28 pc=0x40fe14
runtime.mallocgc(0x400, 0x7a8960, 0x1)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/malloc.go:1002 +0xc9 fp=0xc000645db8 sp=0xc000645d50 pc=0x40f649
runtime.newobject(0xc6?)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/malloc.go:1324 +0x25 fp=0xc000645de0 sp=0xc000645db8 pc=0x40ff05
runtime_test.deferHeapAndStack(0xc7)
	/usr/local/google/home/mknyszek/work/go-1/src/runtime/stack_test.go:924 +0x165 fp=0xc000645ec0 sp=0xc000645de0 pc=0x75bb05

@mknyszek
Copy link
Contributor Author

mknyszek commented Apr 21, 2023

The BP is bad upon entrance to newstack when we fail in the "trace-in-preemption-at-traceGCMarkAssistDone" failure. So it's getting messed up somewhere between the stack switch back after gcAssistAlloc1 and the next preemption.

The thing is, I think this means it has to be morestack putting a bad value into gp.sched.bp, since that's the last piece of code that would write to it. That doesn't mean it produced the bad value, it just means it's producing it. EDIT: I meant "it just means it's writing it," as in "it's passing it on." :P

Indeed, if I print getcallerfp after we switch back stacks (post gcAssistAlloc), it has the wrong value! That suggests to me RBP is somehow set wrong by the return from systemstack. EDIT: I just confirmed also that just before returning from the systemstack closure gp.sched.bp is correct.

@ianlancetaylor
Copy link
Contributor

That doesn't mean it produced the bad value, it just means it's producing it.

Hmmm. Tell us more.

@mknyszek
Copy link
Contributor Author

mknyszek commented Apr 21, 2023

With Cherry's help I think we've got it. The problem is that systemstack is setting the correct BP, but on return BP is set to *BP. That BP comes from the systemstack frame but notably that is skipped by adjustframe. So we have a stale BP on the stack.

EDIT: This is the reason by debugCheckBP wasn't finding the issue. It was being skipped in this one very particular case. :)

@mknyszek
Copy link
Contributor Author

Yeah, looks like the fact that the systemstack isn't adjusted is the problem. This diff causes the problem to no longer reproduce:

diff --git a/src/runtime/stack.go b/src/runtime/stack.go
index e1e6c7e82a..e9c5605377 100644
--- a/src/runtime/stack.go
+++ b/src/runtime/stack.go
@@ -663,12 +663,12 @@ func adjustframe(frame *stkframe, adjinfo *adjustinfo) {
        if stackDebug >= 2 {
                print("    adjusting ", funcname(f), " frame=[", hex(frame.sp), ",", hex(frame.fp), "] pc=", hex(frame.pc), " continpc=", hex(frame.continpc), "\n")
        }
-       if f.funcID == funcID_systemstack_switch {
-               // A special routine at the bottom of stack of a goroutine that does a systemstack call.
-               // We will allow it to be copied even though we don't
-               // have full GC info for it (because it is written in asm).
-               return
-       }
+       //if f.funcID == funcID_systemstack_switch {
+       // A special routine at the bottom of stack of a goroutine that does a systemstack call.
+       // We will allow it to be copied even though we don't
+       // have full GC info for it (because it is written in asm).
+       //      return
+       //}

        locals, args, objs := frame.getStackMap(&adjinfo.cache, true)

@mknyszek
Copy link
Contributor Author

mknyszek commented Apr 21, 2023

Cherry and I think that something like the above diff is actually the real fix. The systemstack frame is faking its PC as systemstack_switch, and although like the comment says it doesn't have GC info for it, there's also never going to be anything interesting in that frame.

Probably the fix looks like getting rid of this case and writing a long doc comment about why it's safe to let adjustframe run on the systemstack frame. Another possible fix is to just move the frame pointer update in adjustframe up above the short-circuit for the systemstack frame, which may be safer just in case there's something we're missing here.

@felixge
Copy link
Contributor

felixge commented Apr 22, 2023

I was also hacking on this last night (mostly using lots of dlog() and stackPoisonCopy = true) and came to the same conclusion that shrinkstack was involved. E.g. the following passes:

while GODEBUG=gcshrinkstackoff=1 ../bin/go test -v -trace /dev/null -run TestDeferHeapAndStack runtime; do :; done

I also agree with the conclusion from @mknyszek and @cherrymui that it's the interaction with systemstack that is problematic.

AFAICT this is a regression from CL 472195.

One thing that still surprised me was that this issue is only observed during stack shrinking and not stack growth. I think the answer to this is that stack growth can't happen while we're on the g0 stack, but stack shrinking (via GC) can happen to a goroutine that switched to g0. Does that sound right?

Another possible fix is to just move the frame pointer update in adjustframe up above the short-circuit for the systemstack frame, which may be safer just in case there's something we're missing here.

+1 - this sounds like the best approach to me.

Unless somebody is already working on it (please comment), I'll try to work on a patch for this.

@mknyszek
Copy link
Contributor Author

One thing that still surprised me was that this issue is only observed during stack shrinking and not stack growth. I think the answer to this is that stack growth can't happen while we're on the g0 stack, but stack shrinking (via GC) can happen to a goroutine that switched to g0. Does that sound right?

That's right. You specifically need a goroutine stack to move while that goroutine has switched to g0; that's the only case where you can fail to update a frame pointer (specifically this systemstack one). And currently the only way for a stack to move while executing on the system stack is if some other goroutine scans the stack and decides to shrink it as well.

@felixge
Copy link
Contributor

felixge commented Apr 24, 2023

Deleted my comment, it was slightly inaccurate. I'll provide more details when I send my patches later today or tomorrow.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/489015 mentions this issue: runtime: fix bug by adjusting systemstack frame

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/489117 mentions this issue: Revert "runtime: turn frame pointer unwinding for tracing off by default"

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/489115 mentions this issue: runtime: add test for systemstack frame pointer adjustment

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/489116 mentions this issue: runtime: remove systemstack logic from adjustframe

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/489016 mentions this issue: runtime: adjust frame pointer of first g0 frame for amd64

@felixge
Copy link
Contributor

felixge commented Apr 26, 2023

The patches above should fix the issue and add better test coverage. The last patch fixes a related issue, but it's not impacting traceEvent unwinding. Details are in the CLs.

gopherbot pushed a commit that referenced this issue May 3, 2023
Add TestSystemstackFramePointerAdjust as a regression test for CL
489015.

By turning stackPoisonCopy into a var instead of const and introducing
the ShrinkStackAndVerifyFramePointers() helper function, we are able to
trigger the exact combination of events that can crash traceEvent() if
systemstack restores a frame pointer that is pointing into the old
stack.

Updates #59692

Change-Id: I60fc6940638077e3b60a81d923b5f5b4f6d8a44c
Reviewed-on: https://go-review.googlesource.com/c/go/+/489115
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Felix Geisendörfer <felix.geisendoerfer@datadoghq.com>
gopherbot pushed a commit that referenced this issue May 3, 2023
Remove logic for skipping some adjustframe logic for systemstack (aka
FuncID_systemstack_switch). This was introduced in 2014 by
9198ed4 but doesn't seem to be needed
anymore.

Updates #59692

Change-Id: I2368d64f9bb28ced4e7f15c9b15dac7a29194389
Reviewed-on: https://go-review.googlesource.com/c/go/+/489116
Reviewed-by: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Felix Geisendörfer <felix.geisendoerfer@datadoghq.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
@golang golang locked and limited conversation to collaborators Apr 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

7 participants