os/signal: test flaky on darwin

[bcmills]

The os/signal test seems to time out fairly frequently on the darwin-amd64-race builder.
It's not obvious to me whether the test is deadlocked or just slow.

Example: https://build.golang.org/log/7c20aa477d782f38c6184e7f8bce05c9648b57b3

panic: test timed out after 3m0s

goroutine 20 [running]:
testing.(*M).startAlarm.func1()
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/testing/testing.go:1332 +0x11c
created by time.goFunc
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/time/sleep.go:169 +0x52

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0000c0000, 0x42154f9, 0x9, 0x421e888, 0x1)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/testing/testing.go:917 +0x68a
testing.runTests.func1(0xc0000c0000)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/testing/testing.go:1157 +0xa7
testing.tRunner(0xc0000c0000, 0xc000046db0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/testing/testing.go:865 +0x163
testing.runTests(0xc00000e060, 0x435ace0, 0xb, 0xb, 0x0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/testing/testing.go:1155 +0x522
testing.(*M).Run(0xc0000ba000, 0x0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/testing/testing.go:1072 +0x2ea
main.main()
	_testmain.go:64 +0x21f

goroutine 4 [syscall]:
os/signal.signal_recv(0x4247580)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/sigqueue.go:139 +0x9f
os/signal.loop()
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/os/signal/signal_unix.go:23 +0x30
created by os/signal.init.0
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/os/signal/signal_unix.go:29 +0x4f

goroutine 10 [runnable]:
runtime.Gosched(...)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/proc.go:266
os/signal.signalWaitUntilIdle()
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/sigqueue.go:172 +0x31
os/signal.Stop(0xc00009c300)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/os/signal/signal.go:196 +0x2f3
runtime.Goexit()
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/panic.go:503 +0xec
testing.(*common).FailNow(0xc0000c0300)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/testing/testing.go:609 +0x5b
testing.(*common).Fatalf(0xc0000c0300, 0x4218523, 0x16, 0xc0000e6d18, 0x1, 0x1)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/testing/testing.go:672 +0x9f
os/signal.waitSig(0xc0000c0300, 0xc00009c2a0, 0x4247580, 0x42436a8)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/os/signal/signal_test.go:32 +0x309
os/signal.testCancel(0xc0000c0300, 0xc00003b700)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/os/signal/signal_test.go:133 +0x1f9
os/signal.TestReset(0xc0000c0300)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/os/signal/signal_test.go:187 +0x3e
testing.tRunner(0xc0000c0300, 0x421e888)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/testing/testing.go:865 +0x163
created by testing.(*T).Run
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/testing/testing.go:916 +0x652

goroutine 8 [runnable]:
runtime.Gosched(...)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/proc.go:266
os/signal.signalWaitUntilIdle()
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/sigqueue.go:172 +0x31
os/signal.Stop(0xc00009c180)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/os/signal/signal.go:196 +0x2f3
os/signal.TestStress.func1(0xc000022480, 0xc0000224e0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/os/signal/signal_test.go:92 +0x194
created by os/signal.TestStress
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/os/signal/signal_test.go:79 +0xe1
FAIL	os/signal	180.028s

[bcmills]

CC @ianlancetaylor for os/signal

[ianlancetaylor]

In the stack trace two goroutines are stuck in this loop in signalWaitUntilIdle (in runtime/sigqueue.go):

	for atomic.Load(&sig.delivering) != 0 {
		Gosched()
	}

sig.delivering can only be non-zero while executing sigsend, also in runtime/sigqueue.go. sigsend is invoked by the signal handler. There is no code in sigsend that blocks, and sigsend always returns with sig.delivering unchanged.

So this seems to be impossible. The failure presumably has something to do with the race detector, but I don't know why it would be Darwin specific.

[bradfitz]

@dvyukov for any ideas.

[bcmills]

Some thoughts:

• It could be another manifestation of the race in runtime: signal handling: sighandler crash due to race with signal.Ignore #20748.
• Perhaps the fact that it's Darwin-specific has something to do with the interaction with libc. (Do we invoke sigaction via libc on Darwin now?) I seem to recall that TSAN tends to defer signal delivery until the next libc call (see runtime: async signals not reliably delivered to Go threads under TSAN #18717), so if we're hitting the TSAN hooks for signal delivery and then the test is idling in a way that doesn't ping back into libc, that could deadlock it.

[bradfitz]

Not just race, it turns out. @andybons notes that it's also here:

https://build.golang.org/log/65130fd6cf64710896cae0a569087f8fa8df66fc (darwin-amd64-10_14)

[bradfitz]

And that last one is on the Go 1.12 release branch.

@randall77, @ianlancetaylor, should this block the Go 1.12.2 release?

[ianlancetaylor]

It's hard to believe that this is new in 1.12.2 compared to 1.12, so I don't see a reason to block the 1.12.2 release.

[rsc]

Just saw this on my pre-Mojave Mac as well. It's too bad the stack trace does not include any signal-handling thread stacks. It looks for all the world like sig.delivering is non-zero and it would be very nice to see where exactly the signal-handling thread is.

panic: test timed out after 9m0s

goroutine 24 [running]:
testing.(*M).startAlarm.func1()
	/Users/rsc/go/src/testing/testing.go:1380 +0xdf
created by time.goFunc
	/Users/rsc/go/src/time/sleep.go:169 +0x44

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000152100, 0x4167a64, 0x9, 0x4170e60, 0x4078a01)
	/Users/rsc/go/src/testing/testing.go:964 +0x377
testing.runTests.func1(0xc0000fe000)
	/Users/rsc/go/src/testing/testing.go:1205 +0x78
testing.tRunner(0xc0000fe000, 0xc000054e30)
	/Users/rsc/go/src/testing/testing.go:912 +0xbf
testing.runTests(0xc0000d4020, 0x426fca0, 0xb, 0xb, 0x0)
	/Users/rsc/go/src/testing/testing.go:1203 +0x2a7
testing.(*M).Run(0xc0000fa000, 0x0)
	/Users/rsc/go/src/testing/testing.go:1120 +0x160
main.main()
	_testmain.go:64 +0x135

goroutine 34 [syscall]:
os/signal.signal_recv(0x41931e0)
	/Users/rsc/go/src/runtime/sigqueue.go:139 +0x9f
os/signal.loop()
	/Users/rsc/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/Users/rsc/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 23 [runnable]:
runtime.Gosched(...)
	/Users/rsc/go/src/runtime/proc.go:269
os/signal.signalWaitUntilIdle()
	/Users/rsc/go/src/runtime/sigqueue.go:172 +0x30
os/signal.Stop(0xc0000be240)
	/Users/rsc/go/src/os/signal/signal.go:196 +0x18c
runtime.Goexit()
	/Users/rsc/go/src/runtime/panic.go:503 +0xec
testing.(*common).FailNow(0xc000152100)
	/Users/rsc/go/src/testing/testing.go:656 +0x39
testing.(*common).Fatalf(0xc000152100, 0x416ab68, 0x16, 0xc000043588, 0x1, 0x1)
	/Users/rsc/go/src/testing/testing.go:719 +0x90
os/signal.waitSig(0xc000152100, 0xc0000be1e0, 0x41931e0, 0x418f4e0)
	/Users/rsc/go/src/os/signal/signal_test.go:32 +0x247
os/signal.testCancel(0xc000152100, 0x2be993261bf00)
	/Users/rsc/go/src/os/signal/signal_test.go:133 +0x1ce
os/signal.TestReset(0xc000152100)
	/Users/rsc/go/src/os/signal/signal_test.go:187 +0x30
testing.tRunner(0xc000152100, 0x4170e60)
	/Users/rsc/go/src/testing/testing.go:912 +0xbf
created by testing.(*T).Run
	/Users/rsc/go/src/testing/testing.go:963 +0x350

goroutine 21 [runnable]:
runtime.Gosched(...)
	/Users/rsc/go/src/runtime/proc.go:269
os/signal.signalWaitUntilIdle()
	/Users/rsc/go/src/runtime/sigqueue.go:172 +0x30
os/signal.Stop(0xc000024120)
	/Users/rsc/go/src/os/signal/signal.go:196 +0x18c
os/signal.TestStress.func1(0xc000138120, 0xc000138180)
	/Users/rsc/go/src/os/signal/signal_test.go:92 +0x160
created by os/signal.TestStress
	/Users/rsc/go/src/os/signal/signal_test.go:79 +0xd3
FAIL	os/signal	540.197s

[bcmills]

One on darwin-arm64-corellium (https://build.golang.org/log/89688609340e1b045e263677ed0bf9b0cdd84710):

Details

panic: test timed out after 3m0s

goroutine 10 [running]:
testing.(*M).startAlarm.func1()
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/testing/testing.go:1382 +0xc0
created by time.goFunc
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/time/sleep.go:169 +0x38

goroutine 1 [chan receive]:
testing.(*T).Run(0x130112200, 0x10248ee96, 0x9, 0x1024981c0, 0x5ce80e01)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/testing/testing.go:961 +0x2b4
testing.runTests.func1(0x1300f4000)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/testing/testing.go:1207 +0x6c
testing.tRunner(0x1300f4000, 0x13003edf8)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/testing/testing.go:909 +0xa8
testing.runTests(0x1300b8060, 0x102596e20, 0xb, 0xb, 0x0)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/testing/testing.go:1205 +0x24c
testing.(*M).Run(0x1300e0000, 0x0)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/testing/testing.go:1122 +0x140
main.main()
	_testmain.go:66 +0x144

goroutine 19 [syscall]:
os/signal.signal_recv(0x1024bbc60)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/runtime/sigqueue.go:139 +0xb8
os/signal.loop()
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/os/signal/signal_unix.go:23 +0x18
created by os/signal.init.0
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/os/signal/signal_unix.go:29 +0x30

goroutine 9 [runnable]:
runtime.Gosched(...)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/runtime/proc.go:269
os/signal.signalWaitUntilIdle()
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/runtime/sigqueue.go:172 +0x2c
os/signal.Stop(0x130080360)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/os/signal/signal.go:196 +0x1a8
runtime.Goexit()
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/runtime/panic.go:503 +0xdc
testing.(*common).FailNow(0x130112200)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/testing/testing.go:653 +0x30
testing.(*common).Fatalf(0x130112200, 0x102491f08, 0x16, 0x13003dd78, 0x1, 0x1)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/testing/testing.go:716 +0x70
os/signal.waitSig(0x130112200, 0x130080300, 0x1024bbc60, 0x1024b7e80)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/os/signal/signal_test.go:32 +0x19c
os/signal.testCancel(0x130112200, 0x13003df00)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/os/signal/signal_test.go:133 +0x188
os/signal.TestReset(0x130112200)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/os/signal/signal_test.go:187 +0x24
testing.tRunner(0x130112200, 0x1024981c0)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/testing/testing.go:909 +0xa8
created by testing.(*T).Run
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/testing/testing.go:960 +0x298

goroutine 7 [runnable]:
runtime.Gosched(...)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/runtime/proc.go:269
os/signal.signalWaitUntilIdle()
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/runtime/sigqueue.go:172 +0x2c
os/signal.Stop(0x1300ca480)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/os/signal/signal.go:196 +0x1a8
os/signal.TestStress.func1(0x130128120, 0x130128180)
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/os/signal/signal_test.go:92 +0x110
created by os/signal.TestStress
	/tmp/workdir-host-darwin-arm64-corellium-ios/go/src/os/signal/signal_test.go:79 +0xd0
FAIL	os/signal	180.141s