os/signal: timeouts in TestNotifyContextNotifications on solaris-amd64-oraclerel #51054

bcmills · 2022-02-07T19:52:53Z

#!watchflakes
post <- pkg == "os/signal" && test == "TestNotifyContextNotifications"

goroutine 73 [syscall]:
runtime.syscall_wait4(0x4317, 0xc00022247c, 0x0, 0xc000260990)
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/runtime/syscall_solaris.go:312 +0x86 fp=0xc0000bbd38 sp=0xc0000bbce0 pc=0x462026
syscall.Wait4(0x203000?, 0xc000204750?, 0xc000229608?, 0x40c945?)
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/syscall/syscall_solaris.go:247 +0x2c fp=0xc0000bbd80 sp=0xc0000bbd38 pc=0x46e2cc
os.(*Process).wait(0xc000248330)
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/os/exec_unix.go:43 +0x85 fp=0xc0000bbdd8 sp=0xc0000bbd80 pc=0x4832a5
os.(*Process).Wait(...)
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/os/exec.go:132
os/exec.(*Cmd).Wait(0xc00025edc0)
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/os/exec/exec.go:507 +0x54 fp=0xc0000bbe50 sp=0xc0000bbdd8 pc=0x4c7a94
os/exec.(*Cmd).Run(0x5?)
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/os/exec/exec.go:341 +0x39 fp=0xc0000bbe70 sp=0xc0000bbe50 pc=0x4c6bf9
os/exec.(*Cmd).CombinedOutput(0xc00025edc0)
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/os/exec/exec.go:567 +0x96 fp=0xc0000bbe98 sp=0xc0000bbe70 pc=0x4c7db6
os/signal.TestNotifyContextNotifications.func2(0xc000278ea0)
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/os/signal/signal_test.go:749 +0x191 fp=0xc0000bbf70 sp=0xc0000bbe98 pc=0x4cf871
testing.tRunner(0xc000278ea0, 0xc0002184f0)
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/testing/testing.go:1440 +0x102 fp=0xc0000bbfc0 sp=0xc0000bbf70 pc=0x4bb6c2
testing.(*T).Run.func1()
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/testing/testing.go:1487 +0x2a fp=0xc0000bbfe0 sp=0xc0000bbfc0 pc=0x4bc56a
runtime.goexit()
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000bbfe8 sp=0xc0000bbfe0 pc=0x4653a1
created by testing.(*T).Run
	/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/testing/testing.go:1487 +0x35f

greplogs --dashboard -md -l -e 'panic: test timed out .*\n(?:\ngoroutine \d+ .*:\n(.+\n)*)*^os/signal\.TestNotifyContextNotifications' --since=2021-01-01

This is the same test that was observed to be flaky in #41561 (CC @henvic, @ianlancetaylor, @cherrymui).

The text was updated successfully, but these errors were encountered:

bcmills · 2022-02-07T19:57:09Z

It's interesting that the test is blocked on this subprocess in particular, because we explicitly plumb in a value for the -test.timeout flag that should be short enough to complete before the test finishes:
https://cs.opensource.google/go/go/+/master:src/os/signal/signal_test.go;l=746-748;drc=e3eaedb5cf623d0836533573db4140749da42768

henvic · 2022-02-08T04:42:40Z

Thanks for pinging me, @bcmills.
I'll try to reproduce it on a Solaris VM during this weekend.

bcmills · 2022-03-08T20:55:47Z

One more this month:

greplogs --dashboard -md -l -e 'panic: test timed out .*\n(?:\ngoroutine \d+ .*:\n(.+\n)*)*^os/signal\.TestNotifyContextNotifications' --since=2022-02-08

2022-03-07T18:19:44-7dc6c5e/solaris-amd64-oraclerel

I notice that we're not actually calling t.Parallel() in the subtests. I wonder if the test is actually timing out in the first subtest, and then starting the second subtest without actually having enough time left to clean up.

It occurs to me that the os/exec changes proposed in #50436 would probably help here too: if either subtest times out, we ought to terminate the subprocess with SIGQUIT to figure out what it's stuck on.

henvic · 2022-03-13T22:33:22Z

* t.Parallel() is called in TestNotifyContextNotifications immediately after the checkNotifyContext conditional.

I was able to reproduce this (building checked out @ commit 0261fa6) when running all.bash on a SunOS solaris 5.11 11.4.0.15.0 i86pc i386 i86pc virtual machine. In my case, it's happening about once every ~20 runs or so. I wasn't able to identify cause. Weirdly, increasing the memory of the virtual machine seems to have reduced the number of times this happened.

bcmills · 2022-03-30T21:06:14Z

I wasn't able to identify cause. Weirdly, increasing the memory of the virtual machine seems to have reduced the number of times this happened.

Since solaris/amd64 is not a first class port, if we can't identify the cause of the failure it would be ok to skip the test on that platform.

greplogs --dashboard -md -l -e 'panic: test timed out .*\n(?:\ngoroutine \d+ .*:\n(.+\n)*)*^os/signal\.TestNotifyContextNotifications' --since=2022-03-08

2022-03-28T19:46:07-1a9972e/solaris-amd64-oraclerel

gopherbot · 2022-04-05T16:16:22Z

Change https://go.dev/cl/398454 mentions this issue: os/signal: run TestNotifyContextnotifications subtests in parallel

gopherbot · 2022-04-05T16:17:00Z

Change https://go.dev/cl/398454 mentions this issue: os/signal: run TestNotifyContextNotifications subtests in parallel

If we run out of time on the first subtest, we don't want to start the second one with essentially no time remaining. (Moreover, there is no compelling reason not to run these tests in parallel, since they send signals to separate processes.) For #51054. Change-Id: I0424e08c3a9d2db986568d5a5c004859b52f7c51 Reviewed-on: https://go-review.googlesource.com/c/go/+/398454 Trust: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Henrique Vicente de Oliveira Pinto <henriquevicente@gmail.com> TryBot-Result: Gopher Robot <gobot@golang.org>

bcmills · 2022-04-05T17:54:18Z

CL 398454 might fix this. I'll keep an eye on it.

bcmills · 2022-04-19T16:22:16Z

The thrilling conclusion, from https://build.golang.org/log/efc839b64c4b5ef269ecf21fa28685fbeb600e18:

--- FAIL: TestNotifyContextNotifications (0.00s)
    --- FAIL: TestNotifyContextNotifications/once (0.00s)
        signal_test.go:741: starting test with less than 22s remaining
    --- FAIL: TestNotifyContextNotifications/multiple (0.00s)
        signal_test.go:741: starting test with less than 22s remaining
FAIL
FAIL	os/signal	158.438s

The large settleTime that we had to set on solaris is just too slow. I guess for the interim we can try scaling it back, but longer term we should rework the tests not to send signals to the main test process to begin with. 🤔

gopherbot · 2022-04-29T20:58:38Z

Change https://go.dev/cl/403199 mentions this issue: os/signal: scale back the solaris-amd64-oraclerel settle time

The settleTime is arbitrary. Ideally we should refactor the test to avoid it (using subprocesses instead of sleeps to isolate tests from each others' delayed signals), but as a shorter-term workaround let's try scaling it back to match linux/ppc64 (the other builder that empirically requires a longer settleTime). For #51054. Updates #33174. Change-Id: I574fffaadd74c52c13d63974e87f20b6d3cf3c4b Reviewed-on: https://go-review.googlesource.com/c/go/+/403199 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Bryan Mills <bcmills@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Run-TryBot: Bryan Mills <bcmills@google.com>

ianlancetaylor · 2022-06-24T20:40:55Z

Rolling forward to 1.20. Please comment if you disagree. Thanks.

gopherbot · 2023-01-17T14:36:14Z

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "os/signal" && test == "TestNotifyContextNotifications"

2023-01-09 17:35 solaris-amd64-oraclerel go@87105e5b os/signal.TestNotifyContextNotifications (log)

panic: test timed out after 3m0s

runtime.gopark(0xc00011bd68?, 0xc00011bdd0?, 0xe5?, 0xc8?, 0xc00011bdf8?)
	/opt/golang/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/runtime/proc.go:361 +0xd6 fp=0xc00011bd50 sp=0xc00011bd30 pc=0x436d56
runtime.chanrecv(0xc0000b0690, 0xc00011be4f, 0x1)
	/opt/golang/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/runtime/chan.go:577 +0x56c fp=0xc00011bde0 sp=0xc00011bd50 pc=0x405f2c
runtime.chanrecv1(0xc000098030?, 0x1?)
	/opt/golang/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/runtime/chan.go:440 +0x18 fp=0xc00011be08 sp=0xc00011bde0 pc=0x405958
testing.(*T).Run(0xc0000036c0, {0x5385d4?, 0x2?}, 0xc000092400)
	/opt/golang/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/testing/testing.go:1487 +0x37a fp=0xc00011bea8 sp=0xc00011be08 pc=0x4bc85a
os/signal.TestNotifyContextNotifications(0xc0001e4340?)
	/opt/golang/tmp/workdir-host-solaris-oracle-amd64-oraclerel/go/src/os/signal/signal_test.go:733 +0x1d1 fp=0xc00011bf70 sp=0xc00011bea8 pc=0x4cfb11
testing.tRunner(0xc0000036c0, 0x543928)

— watchflakes

gopherbot · 2023-02-17T15:09:21Z

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

bcmills · 2023-05-12T14:51:42Z

Duplicate of #60152

bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 7, 2022

bcmills added this to the Go1.19 milestone Feb 7, 2022

bcmills changed the title ~~os/signal: timeout in TestNotifyContextNotifications on solaris-amd64-oraclerel~~ os/signal: timeouts in TestNotifyContextNotifications on solaris-amd64-oraclerel Feb 7, 2022

bcmills added the OS-Solaris label Feb 7, 2022

bcmills mentioned this issue Mar 30, 2022

os/signal: timeouts in TestTime on solaris-amd64-oraclerel #52050

Closed

bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 5, 2022

bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 19, 2022

bcmills mentioned this issue May 2, 2022

x/tools/go/packages: test timouts on solaris-amd64-oraclerel builder #52658

Closed

ianlancetaylor modified the milestones: Go1.19, Go1.20 Jun 24, 2022

bcmills mentioned this issue Jan 17, 2023

os/signal: TestNotifyContextNotifications failures #57829

Closed

bcmills modified the milestones: Go1.20, Backlog Jan 17, 2023

bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jan 17, 2023

gopherbot closed this as completed Feb 17, 2023

bcmills reopened this Feb 17, 2023

gopherbot closed this as completed Feb 17, 2023

bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Feb 17, 2023

bcmills reopened this Feb 17, 2023

bcmills marked this as a duplicate of #60152 May 12, 2023

bcmills closed this as not planned Won't fix, can't repro, duplicate, stale May 12, 2023

golang locked and limited conversation to collaborators May 11, 2024

gopherbot added the FrozenDueToAge label May 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os/signal: timeouts in TestNotifyContextNotifications on solaris-amd64-oraclerel #51054

os/signal: timeouts in TestNotifyContextNotifications on solaris-amd64-oraclerel #51054

bcmills commented Feb 7, 2022 •

edited

Loading

bcmills commented Feb 7, 2022

henvic commented Feb 8, 2022

bcmills commented Mar 8, 2022

henvic commented Mar 13, 2022

bcmills commented Mar 30, 2022

gopherbot commented Apr 5, 2022

gopherbot commented Apr 5, 2022

bcmills commented Apr 5, 2022

bcmills commented Apr 19, 2022

gopherbot commented Apr 29, 2022

ianlancetaylor commented Jun 24, 2022

gopherbot commented Jan 17, 2023

gopherbot commented Feb 17, 2023

bcmills commented May 12, 2023

os/signal: timeouts in TestNotifyContextNotifications on solaris-amd64-oraclerel #51054

os/signal: timeouts in TestNotifyContextNotifications on solaris-amd64-oraclerel #51054

Comments

bcmills commented Feb 7, 2022 • edited Loading

bcmills commented Feb 7, 2022

henvic commented Feb 8, 2022

bcmills commented Mar 8, 2022

henvic commented Mar 13, 2022

bcmills commented Mar 30, 2022

gopherbot commented Apr 5, 2022

gopherbot commented Apr 5, 2022

bcmills commented Apr 5, 2022

bcmills commented Apr 19, 2022

gopherbot commented Apr 29, 2022

ianlancetaylor commented Jun 24, 2022

gopherbot commented Jan 17, 2023

gopherbot commented Feb 17, 2023

bcmills commented May 12, 2023

bcmills commented Feb 7, 2022 •

edited

Loading