runtime: "fatal error: runtime: g is running but p is not"

[gpaul]

Our benchmarks hit this overnight. It happened only once. Reproducing it is proving
difficult so I'm filing what I have so far.

Details:
go version devel +9169cb38c3e8 Tue Oct 22 18:33:37 2013 -0400 linux/amd64
Linux foobox 3.4.7-1.fc16.x86_64
GOMAXPROCS=8

The benchmarks are run by a go binary that runs the codec4.test using os/exec. The
results and stacktrace follows

PASS
...
BenchmarkCodec4Uint16SEnumTupleProtoMarshal-8 fatal error: runtime: g is running but p
is not

runtime stack:
runtime.throw(0xbc0c89)
/build/go/go/src/pkg/runtime/panic.c:464 +0x69
runtime.newstack()
/build/go/go/src/pkg/runtime/stack.c:259 +0x6de
runtime.morestack()
/build/go/go/src/pkg/runtime/asm_amd64.s:225 +0x61

goroutine 28 [stack split]:
acquirep(0xc210010000)
/build/go/go/src/pkg/runtime/proc.c:2269 fp=0x7f732208edc0
procresize(0x8)
/build/go/go/src/pkg/runtime/proc.c:2258 +0x30a fp=0x7f732208edf8
runtime.starttheworld()
/build/go/go/src/pkg/runtime/proc.c:508 +0x235 fp=0x7f732208ee38
runtime.ReadMemStats(0xbcfe20)
/build/go/go/src/pkg/runtime/mgc0.c:2199 +0x9b fp=0x7f732208ee50
testing.(*B).StopTimer(0xc210299690)
/build/go/go/src/pkg/testing/benchmark.go:70 +0x95 fp=0x7f732208ee90
next/gogoprotobuf/test/codec4.BenchmarkCodec4Uint16SEnumTupleProtoMarshal(0xc210299690)
/build/next/next/src/next/gogoprotobuf/test/codec4/codec4pb_test.go:445 +0x19e
fp=0x7f732208ef18
testing.(*B).runN(0xc210299690, 0xf4240)
/build/go/go/src/pkg/testing/benchmark.go:119 +0x88 fp=0x7f732208ef28
testing.(*B).launch(0xc210299690)
/build/go/go/src/pkg/testing/benchmark.go:207 +0x148 fp=0x7f732208ef98
runtime.goexit()
/build/go/go/src/pkg/runtime/proc.c:1394 fp=0x7f732208efa0
created by testing.(*B).run
/build/go/go/src/pkg/testing/benchmark.go:170 +0x2e

goroutine 1 [chan receive]:
testing.(*B).run(0xc210299690, 0x3, 0x7f732207ed80, 0x1, 0x1, ...)
/build/go/go/src/pkg/testing/benchmark.go:171 +0x4b
testing.RunBenchmarks(0x815d98, 0xbc1340, 0x28, 0x28)
/build/go/go/src/pkg/testing/benchmark.go:303 +0x549
testing.Main(0x815d98, 0xbc2be0, 0x51, 0x51, 0xbc1340, ...)
/build/go/go/src/pkg/testing/testing.go:409 +0x1b5
main.main()
next/gogoprotobuf/test/codec4/_test/_testmain.go:287 +0x9c

goroutine 4 [finalizer wait]:
runtime.park(0x40afc0, 0xbc4690, 0xbc24c8)
/build/go/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
/build/go/go/src/pkg/runtime/mgc0.c:2276 +0x84
runtime.goexit()
/build/go/go/src/pkg/runtime/proc.c:1394

[adg]

Comment 1:

Looks bad. Note that 9169cb38c3e8 is essentially tip.

Labels changed: added priority-asap, removed priority-triage.

Status changed to Accepted.

[adg]

Comment 2:

Labels changed: added go1.2.

[gpaul]

Comment 3:

I've been able to reproduce the issue but I'm still trimming the code. I'll post an
update soon.

[rsc]

Comment 4:

This is almost certainly the fix:
diff -r 9169cb38c3e8 src/pkg/runtime/stack.c
--- a/src/pkg/runtime/stack.c   Tue Oct 22 18:33:37 2013 -0400
+++ b/src/pkg/runtime/stack.c   Wed Oct 23 09:49:42 2013 -0400
@@ -255,7 +255,7 @@
    if(gp->stackguard0 == (uintptr)StackPreempt) {
        if(gp == m->g0)
            runtime·throw("runtime: preempt g0");
-       if(oldstatus == Grunning && m->p == nil)
+       if(oldstatus == Grunning && m->p == nil && m->locks == 0)
            runtime·throw("runtime: g is running but p is not");
        if(oldstatus == Gsyscall && m->locks == 0)
            runtime·throw("runtime: stack split during syscall");
If you can find a small case that reproduces the problem that we can add as a test case,
great. But actually I'd be more interested just to know whether the change fixes your
reproduction case.
Thanks.

[dvyukov]

Comment 5:

When is it possible that oldstatus == Grunning && m->p == nil && m->locks > 0?

[rsc]

Comment 6:

There is a useful stack trace in the initial report. It is possible when a
goroutine is calling acquirep during starttheworld during ReadMemStats.
Russ

[gpaul]

Comment 7:

I've been looping the tests with your fix and it hasn't reoccurred. I'll leave them
overnight just in case. Reducing the reproduction case without losing the bug is proving
tricky. Thanks for the fix.

[gpaul]

Comment 8:

The tests are still looping and the issue hasn't reappeared. Will your fix make it into
go1.2?

[rsc]

Comment 9:

This issue was closed by revision 00a757f.

Status changed to Fixed.

[adg]

Comment 10:

This issue was closed by revision 0655b706babf.