Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: bad P in runqempty called from checkRunqsNoP #47304

Closed
mknyszek opened this issue Jul 20, 2021 · 3 comments
Closed

runtime: bad P in runqempty called from checkRunqsNoP #47304

mknyszek opened this issue Jul 20, 2021 · 3 comments
Labels
Milestone

Comments

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Jul 20, 2021

In trying to reproduce #47302 I wrote a little program. I failed to reproduce it, but instead I found a new failure.

The program is:

package main

import (
	"runtime"
	"time"
)

var sink []byte
var sink2 []byte

func main() {
	go func() {
		for {
			sink = make([]byte, 1024)
		}
	}()
	go func() {
		for {
			sink2 = make([]byte, 1024)
		}
	}()
	for i := 0; i < 5; i++ {
		time.Sleep(100 * time.Millisecond)
		runtime.GOMAXPROCS(100)
		time.Sleep(100 * time.Millisecond)
		runtime.GOMAXPROCS(4)
		time.Sleep(100 * time.Millisecond)
		runtime.GOMAXPROCS(6)
	}
}

The failure is:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x5f0 pc=0x434824]

runtime stack:
runtime.throw({0x4696a7, 0x100000081})
	/usr/local/google/home/mknyszek/toolchain/src/runtime/panic.go:1198 +0x71
runtime.sigpanic()
	/usr/local/google/home/mknyszek/toolchain/src/runtime/signal_unix.go:719 +0x396
runtime.runqempty(...)
	/usr/local/google/home/mknyszek/toolchain/src/runtime/proc.go:5922
runtime.checkRunqsNoP({0xc000026000, 0x180, 0x4}, {0xc000018058, 0x7ffcc3668c48, 0x41e725})
	/usr/local/google/home/mknyszek/toolchain/src/runtime/proc.go:3091 +0x84
runtime.findrunnable()
	/usr/local/google/home/mknyszek/toolchain/src/runtime/proc.go:2891 +0x425
runtime.schedule()
	/usr/local/google/home/mknyszek/toolchain/src/runtime/proc.go:3367 +0x239
runtime.park_m(0xc000001380)
	/usr/local/google/home/mknyszek/toolchain/src/runtime/proc.go:3516 +0x14d
runtime.mcall()
	/usr/local/google/home/mknyszek/toolchain/src/runtime/asm_amd64.s:307 +0x43

goroutine 1 [sleep]:
time.Sleep(0x5f5e100)
	/usr/local/google/home/mknyszek/toolchain/src/runtime/time.go:193 +0x12e
main.main()
	/usr/local/google/home/mknyszek/tmp/gmp.go:25 +0x5b

goroutine 5 [runnable]:
main.main.func1()
	/usr/local/google/home/mknyszek/tmp/gmp.go:14 +0x28
created by main.main
	/usr/local/google/home/mknyszek/tmp/gmp.go:12 +0x26

goroutine 6 [runnable]:
main.main.func2()
	/usr/local/google/home/mknyszek/tmp/gmp.go:19 +0x28
created by main.main
	/usr/local/google/home/mknyszek/tmp/gmp.go:17 +0x34

I can reproduce this at tip, and going back about a month. It takes about 90 minutes (sometimes more, sometimes less) to reproduce when running under https://pkg.go.dev/golang.org/x/tools/cmd/stress. I don't think this is a release blocker because we've never seen it out in the wild, and this program is very unrealistic. It may also reproduce with older releases, I'm not sure.

@mknyszek mknyszek added this to the Backlog milestone Jul 20, 2021
@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Jul 20, 2021

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 21, 2021

Change https://golang.org/cl/336449 mentions this issue: runtime: move mem profile sampling into m-acquired section

Loading

@gopherbot gopherbot closed this in fdb45ac Jul 22, 2021
@dmitshur dmitshur removed this from the Backlog milestone Jul 26, 2021
@dmitshur dmitshur added this to the Go1.17 milestone Jul 26, 2021
steeve added a commit to znly/go that referenced this issue Aug 19, 2021
It was not safe to do mcache profiling updates outside the critical
section, but we got lucky because the runtime was not preemptible.
Adding chunked memory clearing (CL 270943) created preemption
opportunities, which led to corruption of runtime data structures.

Fixes golang#47304.
Fixes golang#47302.

Change-Id: I461615470d62328a83ccbac537fbdc6dcde81c85
Reviewed-on: https://go-review.googlesource.com/c/go/+/336449
Trust: David Chase <drchase@google.com>
Run-TryBot: David Chase <drchase@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Keith Randall <khr@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 30, 2021

Change https://golang.org/cl/359796 mentions this issue: runtime: add always-preempt maymorestack hook

Loading

gopherbot pushed a commit that referenced this issue Nov 5, 2021
This adds a maymorestack hook that forces a preemption at every
possible cooperative preemption point. This would have helped us catch
several recent preemption-related bugs earlier, including #47302,
 #47304, and #47441.

For #48297.

Change-Id: Ib82c973589c8a7223900e1842913b8591938fb9f
Reviewed-on: https://go-review.googlesource.com/c/go/+/359796
Trust: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: David Chase <drchase@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants