runtime: fatal error: releasep: invalid p state

[mikioh]

On the solaris-amd64-smartos buildbot. See http://build.golang.org/log/3e4aa690c01a4123a508595aa248329b26499b18.

ok      log 0.018s
ok      log/syslog  2.047s
ok      math    0.029s
releasep: m=0xc82044a000 m->p=0xc820022000 p->m=859534827520 m->mcache=0xfffffd7fff131a30 p->mcache=0xfffffd7fff131a30 p->status=3
fatal error: releasep: invalid p state

runtime stack:
runtime.throw(0x634c35, 0x19)
    /root/work/solaris-amd64-smartos-73a0185ad390/go/src/runtime/panic.go:566 +0x8b
runtime.releasep(0x2000)
    /root/work/solaris-amd64-smartos-73a0185ad390/go/src/runtime/proc.go:3365 +0x208
runtime.gcstopm()
    /root/work/solaris-amd64-smartos-73a0185ad390/go/src/runtime/proc.go:1760 +0x42
runtime.schedule()
    /root/work/solaris-amd64-smartos-73a0185ad390/go/src/runtime/proc.go:2070 +0x105
runtime.park_m(0xc82044d380)
    /root/work/solaris-amd64-smartos-73a0185ad390/go/src/runtime/proc.go:2169 +0x11a
runtime.mcall(0x0)
    /root/work/solaris-amd64-smartos-73a0185ad390/go/src/runtime/asm_amd64.s:240 +0x64
(snip)

/CC looks like runtime/GC gatekeepers (who?). It expected _Prunning but got _Pgcstop.

[mikioh]

/CC @aclements (from the dying message invalid p state and #10240)

[davecheney]

I just hit this as well

go version devel +a4650a2 Mon Apr 11 21:38:18 2016 +0000 linux/amd64

# github.com/juju/juju/provider/joyent
releasep: m=0xc820062000 m->p=0xc82001f500 p->m=859530731520 m->mcache=0x7ff2acede4b0 p->mcache=0x7ff2acede4b0 p->status=3
fatal error: releasep: invalid p state

runtime stack:
runtime.throw(0x12ce3b2, 0x19)
        /home/dfc/go/src/runtime/panic.go:566 +0x8b
runtime.releasep(0xc00000001)
        /home/dfc/go/src/runtime/proc.go:3365 +0x208
runtime.gcstopm()
        /home/dfc/go/src/runtime/proc.go:1760 +0x42
runtime.schedule()
        /home/dfc/go/src/runtime/proc.go:2070 +0x105
runtime.park_m(0xc82008ea80)
        /home/dfc/go/src/runtime/proc.go:2169 +0x11a
runtime.mcall(0x0)
        /home/dfc/go/src/runtime/asm_amd64.s:240 +0x5b

This feels like a recent regression.

[davecheney]

I suspect a4650a2, /cc @josharian

[josharian]

Ugh. Will take a look, although today promises to be busy. How reliably can you reproduce? Any hints for doing so? If reliably, can you confirm that it was the append changes?

[aclements]

It's not specific to Solaris. It did start right after a4650a2. It's not immediately obvious to me how that would have caused this, but statistically speaking, it's a likely candidate.

$ greplogs -l -md -dashboard -E 'releasep: invalid p state'
2016-04-11T22:35:22-d1feddb/linux-amd64-clang
2016-04-11T23:15:02-501ddf7/linux-amd64-kube
2016-04-12T01:07:46-73a0185/solaris-amd64-smartos

$ greplogs -l -dashboard -E 'releasep: invalid p state' | findflakes -paths
First observed d1feddb 11 Apr 22:35 2016 (11 commits ago)
Last observed  73a0185 12 Apr 01:07 2016 (6 commits ago)
4.7% chance failure is still happening
40% failure probability (3 of 6 commits)
Likely culprits:
   40% d1feddb cmd/vet: improve documentation for flags, slightly
   24% a4650a2 cmd/compile: avoid write barrier in append fast path
   14% 32efa16 cmd/compile: added stats printing to stackalloc
    9% 7e40627 cmd/compile: zero all three argstorage slots
    5% 7f53391 cmd/compile: fix -N build
No known past failures

[josharian]

I'm puzzled too, although spidey sense says missing VARDEF. When I'm at a computer (or if one of you beats me to it) next, we can temporarily disable the inplace append code with a well-placed && false to get this stable again.

[josharian]

I can't seem to reproduce on darwin, which is going to make this a bear to track down. Mailed CL 21891 to disable in the meantime.

[gopherbot]

CL https://golang.org/cl/21891 mentions this issue.

[aclements]

I'll see if I can reproduce on linux/amd64.