runtime: loop over allp causes a nil pointer dereference crash

[mikioh]

See https://build.golang.org/log/ff2e3f95d35cbaec0291bf387a6409c1f71e417f

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8078907]

runtime stack:
runtime.throw(0x8150279, 0x2a)
	/tmp/workdir/go/src/runtime/panic.go:616 +0x6b
runtime.sigpanic()
	/tmp/workdir/go/src/runtime/signal_unix.go:366 +0x230
runtime.runqempty(0x0, 0xffffff01)
	/tmp/workdir/go/src/runtime/proc.go:4638 +0x17
runtime.findrunnable(0x383e6000, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:2342 +0x18d
runtime.schedule()
	/tmp/workdir/go/src/runtime/proc.go:2530 +0x10b
runtime.mstart1(0x0)
	/tmp/workdir/go/src/runtime/proc.go:1232 +0x7a
runtime.mstart()
	/tmp/workdir/go/src/runtime/proc.go:1188 +0x59

Seems like there is an exception to the assumption that "none of allp[i] are nil or in state _Pdead." in http://golang.org/cl/45575.

Hej Hej, @aclements

[ianlancetaylor]

This is probably still happening.

2017-09-28T21:09:46-a299345/darwin-arm-a1428ios
2017-10-11T13:42:27-4477107/darwin-amd64-10_12
2017-10-20T22:03:07-4e64ee4/freebsd-386-110
2017-10-26T20:38:48-da95254/darwin-386-10_11
2017-11-06T20:17:05-eca28cc/linux-amd64-noopt
2017-11-06T20:21:15-fa62ea6/linux-amd64-clang
2017-11-18T00:33:04-12e2933/freebsd-amd64-gce101
2017-12-12T04:26:40-ddae7fb/freebsd-386-11_1

[ianlancetaylor]

The failures are all in programs that call runtime.GOMAXPROCS to increase the number of P's. The failures in all cases are in the // check all runqueues once again loop in findrunnable. Perhaps of particular interest are three failures running go/test/fixedbugs/issue13160.go, a program that calls runtime.GOMAXPROCS once at the start of main. In one of those (https://build.golang.org/log/e9f4fd250aa74c540edb5e75f824d341f9d8a1e1) the failure is in a call to gopark calling schedule, likely from a blocking channel operation. The code in that case is apparently running well after the call to runtime.GOMAXPROCS has returned. So somehow findrunnable is seeing a nil entry in allp after GOMAXPROCS returns. Which seems impossible.

[aclements]

The failures in all cases are in the // check all runqueues once again loop in findrunnable.

That loop is concerning because it runs (somewhat non-obviously) without a P, so it could be running concurrently with a STW. I'm not sure how that would be a problem in that particular test, since, as you point out, allp should never be changing after the initial GOMAXPROCS.

I have a fix, which I will send shortly. I can't reproduce this issue, so it's hard to say if this actually fixes the problem, but it's certainly not a bad thing to fix.

[gopherbot]

Change https://golang.org/cl/86215 mentions this issue: runtime: avoid race on allp in findrunnable

[ianlancetaylor]

I'm troubled by https://build.golang.org/log/71e1620c62a2255007ea450fc02e9c5d6f5af068 which shows a crash in a different part of findrunnable that refers to allp. In that case we are clearly running with a p, and the entry in allp is not nil, but it still somehow got a nil pointer dereference.

[ianlancetaylor]

Most and perhaps all the recent occurrences of this problem seem to be on mipsle.

2018-04-29T03:33:09-13cd006/linux-mipsle
2018-05-07T13:31:15-7b7a854/linux-mipsle
2018-05-07T15:12:35-9d72e8c/linux-mipsle
2018-05-18T13:54:12-8a16c71/linux-mipsle
2018-05-18T16:01:32-db37e16/linux-mipsle
2018-05-29T17:45:36-db9341a/linux-mipsle
2018-06-05T20:22:07-9be4f31/linux-mipsle
2018-06-12T15:03:46-43a3cdf/linux-mipsle
2018-06-13T14:55:01-a2f72cc/linux-mipsle
2018-06-25T18:59:39-50bd1c4/linux-mipsle
2018-06-26T14:39:02-5fad090/linux-mipsle
2018-06-27T22:04:08-83092a4/linux-mipsle
2018-06-28T23:01:22-b410ce7/linux-mipsle
2018-06-29T21:45:46-cdce824/linux-mipsle

[ianlancetaylor]

Several of these seem to be nil pointer dereferences:

##### ../misc/cgo/test
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x43b498]

runtime stack:
runtime.throw(0x9301d0, 0x2a)
	/data/mipsle/go/src/runtime/panic.go:608 +0x58
runtime.sigpanic()
	/data/mipsle/go/src/runtime/signal_unix.go:374 +0x3b0
runtime.findrunnable(0x1022000, 0x0)
	/data/mipsle/go/src/runtime/proc.go:2349 +0xbc
runtime.schedule()
	/data/mipsle/go/src/runtime/proc.go:2612 +0x154
runtime.park_m(0x1192380)
	/data/mipsle/go/src/runtime/proc.go:2675 +0xb0

proc.go:2349 is

if gp := runqsteal(_p_, allp[enum.position()], stealRunNextG); gp != nil {

It's hard to see a nil dereference there. runqsteal is not being inlined.