runtime: panic fatal error: malloc deadlock #24059
Description
What version of Go are you using (go version)?
I've bisected the issue to 2edc4d4
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
darwin/amd64
What did you do?
During development of a large internal service, we were running our test suite. The tests triggered a nil pointer dereference. On 1.9.4 the output was the usual
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1300854]
But on 1.10 we instead got
panic: fatal error: malloc deadlock
panic during panic
runtime stack:
runtime.startpanic_m()
/Users/dgryski/work/src/go/src/runtime/panic.go:693 +0x179
runtime.systemstack(0x0)
/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
/Users/dgryski/work/src/go/src/runtime/proc.go:1170
goroutine 207 [running]:
runtime.systemstack_switch()
/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:363 fp=0xc420373a88 sp=0xc420373a80 pc=0x1057fe0
runtime.startpanic()
/Users/dgryski/work/src/go/src/runtime/panic.go:592 +0x1e fp=0xc420373aa0 sp=0xc420373a88 pc=0x102c8be
runtime.throw(0x1367aa4, 0xf)
/Users/dgryski/work/src/go/src/runtime/panic.go:618 +0x74 fp=0xc420373ac0 sp=0xc420373aa0 pc=0x102c9e4
runtime.mallocgc(0x2010, 0x0, 0x1303401, 0x12ae3a8)
/Users/dgryski/work/src/go/src/runtime/malloc.go:621 +0x99d fp=0xc420373b60 sp=0xc420373ac0 pc=0x10126cd
runtime.itabAdd(0x2541840)
/Users/dgryski/work/src/go/src/runtime/iface.go:122 +0x83 fp=0xc420373bb0 sp=0xc420373b60 pc=0x100f403
runtime.getitab(0x1303480, 0x12e2680, 0xc4202e4301, 0xc4202e43b0)
/Users/dgryski/work/src/go/src/runtime/iface.go:70 +0x404 fp=0xc420373c28 sp=0xc420373bb0 pc=0x100f284
runtime.assertE2I2(0x1303480, 0x12e2680, 0xc42022ad30, 0x7, 0xc400000007, 0xc420373cb8)
/Users/dgryski/work/src/go/src/runtime/iface.go:592 +0x43 fp=0xc420373c58 sp=0xc420373c28 pc=0x10107e3
runtime.printany(0x12e2680, 0xc42022ad30)
/Users/dgryski/work/src/go/src/runtime/error.go:77 +0x69 fp=0xc420373d18 sp=0xc420373c58 pc=0x1007709
runtime.printpanics(0xc420373da0)
/Users/dgryski/work/src/go/src/runtime/panic.go:420 +0x71 fp=0xc420373d38 sp=0xc420373d18 pc=0x102c101
panic(0x12fd360, 0x1528330)
/Users/dgryski/work/src/go/src/runtime/panic.go:553 +0x3b4 fp=0xc420373dd8 sp=0xc420373d38 pc=0x102c544
runtime.panicmem()
/Users/dgryski/work/src/go/src/runtime/panic.go:63 +0x5e fp=0xc420373df8 sp=0xc420373dd8 pc=0x102b24e
runtime.sigpanic()
/Users/dgryski/work/src/go/src/runtime/signal_unix.go:388 +0x17a fp=0xc420373e48 sp=0xc420373df8 pc=0x1041fca
github.com/fastly/service1/service2.(*GetStats).Add(...)
/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:222
github.com/fastly/service1/service2.(*Client).GetAll.func2()
/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:276 +0x27d fp=0xc420373fe0 sp=0xc420373e48 pc=0x1292c8d
runtime.goexit()
/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420373fe8 sp=0xc420373fe0 pc=0x105ab31
created by github.com/fastly/service1/service2.(*Client).GetAll
/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:285 +0x232
[ output trimmed of all the other goroutine stack traces]
Some notes from trying to minimize this into a useful test case:
- There are lots of goroutines running, and the nil dereference happens in one
- When running with
-test.v, the normalpanicoutput is produced - I need to run multiple tests from this package in addition to the failing test. Running just the one test that fails produced the normal
panicoutput. - Adding
-racedoesn't make a difference