Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: timeout in TestNeedmDeadlock on darwin-amd64-10_12 builder #44303

bcmills opened this issue Feb 16, 2021 · 6 comments

runtime: timeout in TestNeedmDeadlock on darwin-amd64-10_12 builder #44303

bcmills opened this issue Feb 16, 2021 · 6 comments


Copy link

@bcmills bcmills commented Feb 16, 2021


It looks like the testprogcgo process ends up wedged, but runBuildTestProg fails to terminate it so we don't get a useful stack trace.

See previously #42207 (the deadlock for which this test was added).
See also #43625 (TestNeedmDeadlock on MIPS).

goroutine 22881 [syscall, 2 minutes]:
syscall.syscall6(0x108d5e0, 0x19ef, 0xc0003bec74, 0x0, 0xc00007a090, 0x0, 0x0, 0x0, 0x0, 0x0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/sys_darwin.go:41 +0x2e fp=0xc0003bebc0 sp=0xc0003beb98 pc=0x107822e
syscall.wait4(0x19ef, 0xc0003bec74, 0x0, 0xc00007a090, 0x90, 0x1285a20, 0xc0003bed01)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/syscall/zsyscall_darwin_amd64.go:44 +0x85 fp=0xc0003bec38 sp=0xc0003bebc0 pc=0x108a345
syscall.Wait4(0x19ef, 0xc0003becc4, 0x0, 0xc00007a090, 0xc00010e420, 0x60, 0x100fd7b)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/syscall/syscall_bsd.go:144 +0x51 fp=0xc0003bec88 sp=0xc0003bec38 pc=0x1088af1
os.(*Process).wait(0xc000078030, 0xc00010e420, 0x12a5560, 0xc000080480)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/os/exec_unix.go:43 +0x85 fp=0xc0003becf8 sp=0xc0003bec88 pc=0x10a48c5
os/exec.(*Cmd).Wait(0xc0003c8000, 0x12a7ab8, 0xc00010e420)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/os/exec/exec.go:507 +0x65 fp=0xc0003bed78 sp=0xc0003becf8 pc=0x113b785
runtime_test.runBuiltTestProg(0xc000483500, 0xc00013e100, 0x71, 0x1291263, 0xd, 0x0, 0x0, 0x0, 0x0, 0x0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/crash_test.go:103 +0x2ed fp=0xc0003bee70 sp=0xc0003bed78 pc=0x11810cd
runtime_test.runTestProg(0xc000483500, 0x1290916, 0xb, 0x1291263, 0xd, 0x0, 0x0, 0x0, 0xc0321263d0, 0x321263d0010f64d0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/crash_test.go:57 +0x165 fp=0xc0003beef0 sp=0xc0003bee70 pc=0x1180d45
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/crash_cgo_test.go:628 +0x7b fp=0xc0003bef80 sp=0xc0003beef0 pc=0x11809fb
testing.tRunner(0xc000483500, 0x12a7170)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/testing/testing.go:1194 +0xef fp=0xc0003befd0 sp=0xc0003bef80 pc=0x10f6b2f
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0003befd8 sp=0xc0003befd0 pc=0x107b421
created by testing.(*T).Run
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/testing/testing.go:1239 +0x2b3
FAIL	runtime	180.025s

CC @ianlancetaylor @cherrymui

@bcmills bcmills added the NeedsInvestigation label Feb 16, 2021
@bcmills bcmills added this to the Go1.17 milestone Feb 16, 2021
Copy link
Member Author

@bcmills bcmills commented Feb 16, 2021

Tentatively milestoning as a release-blocker for Go 1.17 until we better understand the root cause. This may be a deadlock in the runtime, and the failure mode would make it very difficult for a Go user to debug.

Copy link

@dmitshur dmitshur commented Mar 25, 2021

CC @prattmic.

Copy link

@dmitshur dmitshur commented Apr 8, 2021

Friendly ping after 2 weeks since it's a release blocking issue for Go 1.17.

Copy link

@ianlancetaylor ianlancetaylor commented Apr 15, 2021

There seem to only be two failures since the test was added in October, 2020. Both failures are on Darwin 10.12. In both failures the test hits a 3 minute timeout, which is interesting because

  1. The test program (in runtime/testdata/testprogcgo/needmdeadlock.go) itself has a 5 second timeout before exiting, and
  2. the test runner (runBuiltTestProg in runtime/crash_test.go) has a 1 minute timeout before sending a SIGQUIT signal.

It seems that the test program has deadlocked so hard that both of those timeouts fail to stop it.

Copy link

@ianlancetaylor ianlancetaylor commented Apr 15, 2021

This has only happened twice, it's only happened on macOS 10.12, and we are dropping support for macOS 10.12 in 1.17 anyhow. Removing the release-blocker label.

@ianlancetaylor ianlancetaylor removed this from the Go1.17 milestone Apr 15, 2021
@ianlancetaylor ianlancetaylor added this to the Backlog milestone Apr 15, 2021
Copy link

@gopherbot gopherbot commented Apr 22, 2021

Change mentions this issue: dashboard: limit macOS 10.12 builders to at most go1.16

gopherbot pushed a commit to golang/build that referenced this issue Apr 22, 2021
macOS will not be supported after go1.16. This change limits the macOS
10.12 builders to build previous to go1.17.

Updates golang/go#44303
For golang/go#23011

Change-Id: Ib6b4e7939ffb9032b255574d3f0b4fffaad97803
Trust: Carlos Amedee <>
Run-TryBot: Carlos Amedee <>
TryBot-Result: Go Bot <>
Reviewed-by: Dmitri Shuralyov <>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

4 participants