runtime: TestGdbCoreCrashThreadBacktrace failures

[gopherbot]

#!watchflakes
post <- pkg == "runtime" && test == "TestGdbCoreCrashThreadBacktrace"

Issue created automatically to collect these failures.

Example (log):

--- FAIL: TestGdbCoreCrashThreadBacktrace (7.67s)
    runtime-gdb_test.go:81: gdb version 10.1
    runtime-gdb_unix_test.go:334: child output:
        Setting RLIMIT_CORE = syscall.Rlimit{Cur:0xffffffffffffffff, Max:0xffffffffffffffff}
        SIGSEGV: segmentation violation
        PC=0x48cc65 m=0 sigcode=1 addr=0x0
        signal arrived during cgo execution

        goroutine 1 gp=0xc0000081c0 m=0 mp=0x548740 [syscall]:
        runtime.cgocall(0x48cc6e, 0xc00006ff20)
...
        #19 0x000000000043b076 in runtime.mstart0 () at /workdir/go/src/runtime/proc.go:1684
        #20 0x0000000000462c65 in runtime.mstart () at /workdir/go/src/runtime/asm_amd64.s:394
        #21 0x000000000048d669 in crosscall1 () at gcc_amd64.S:42
        #22 0x0000000000802000 in ?? ()
        #23 0x00007f8b74b75fc0 in ?? ()
        #24 0x00007ffd0d4c302f in ?? ()
        #25 0x000000c000008a80 in ?? ()
        #26 0x0000000000462c60 in ?? ()
        #27 0x0000000000000000 in ?? ()
    runtime-gdb_unix_test.go:375: could not find trigger_crash in backtrace

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "runtime" && test == "TestGdbCoreCrashThreadBacktrace"

2023-12-14 22:24 linux-amd64-bullseye go@d73b4322 runtime.TestGdbCoreCrashThreadBacktrace (log)

--- FAIL: TestGdbCoreCrashThreadBacktrace (7.67s)
    runtime-gdb_test.go:81: gdb version 10.1
    runtime-gdb_unix_test.go:334: child output:
        Setting RLIMIT_CORE = syscall.Rlimit{Cur:0xffffffffffffffff, Max:0xffffffffffffffff}
        SIGSEGV: segmentation violation
        PC=0x48cc65 m=0 sigcode=1 addr=0x0
        signal arrived during cgo execution

        goroutine 1 gp=0xc0000081c0 m=0 mp=0x548740 [syscall]:
        runtime.cgocall(0x48cc6e, 0xc00006ff20)
...
        #19 0x000000000043b076 in runtime.mstart0 () at /workdir/go/src/runtime/proc.go:1684
        #20 0x0000000000462c65 in runtime.mstart () at /workdir/go/src/runtime/asm_amd64.s:394
        #21 0x000000000048d669 in crosscall1 () at gcc_amd64.S:42
        #22 0x0000000000802000 in ?? ()
        #23 0x00007f8b74b75fc0 in ?? ()
        #24 0x00007ffd0d4c302f in ?? ()
        #25 0x000000c000008a80 in ?? ()
        #26 0x0000000000462c60 in ?? ()
        #27 0x0000000000000000 in ?? ()
    runtime-gdb_unix_test.go:375: could not find trigger_crash in backtrace

— watchflakes

[prattmic]

This test is new in https://go.dev/cl/536895. The most obvious cause for this failure would be the crash thread failing to crash for >5s. However, we don't have any timing information in the test log.

cc @zzkcode @cherrymui

[prattmic]

Actually, a better theory: usleep doesn't loop on early return of the nanosleep syscall (e.g., if another signal is delivered), so it could return in <5s. We should really fix that.

[zzkcode]

Hi @prattmic , thanks! I would like to send a CL to fix this.

This test is new in https://go.dev/cl/536895. The most obvious cause for this failure would be the crash thread failing to crash for >5s. However, we don't have any timing information in the test log.

cc @zzkcode @cherrymui

Thanks. Are you saying that we should add more logs about timing in signal_unix.go(maybe with how many ms still waiting SIGQUIT also) in each loop? Since it seems hard to print timing related in this test code except printing the overall time.

Actually, a better theory: usleep doesn't loop on early return of the nanosleep syscall (e.g., if another signal is delivered), so it could return in <5s. We should really fix that.

Sorry, I don't know if I really get this. You are talking about usleep may sleep less than what we want if a signal is delivered, and maybe one of the thread(not the faulting thread which cause crash) may crash the process instead of the faulting thread? So use nanosleep instead?

I guess, in this case, maybe it just really takes more than 5s to deliver the SIGQUIT signals since this case takes 7.67s in running.

--- FAIL: TestGdbCoreCrashThreadBacktrace (7.67s)

However, it still stranges to me that the ms which are in usleep state should either the faulting thread or the thread that had received the SIGQUIT. In the former case, this test should still pass; in the latter, won't they just blocked the signal?

// All m's that have already received SIGQUIT have signal masks blocking
// receipt of any signals, so the SIGQUIT will go to an m that hasn't seen it yet.

[bcmills]

@prattmic, since this test is new as of Go 1.22, should it be marked as a release blocker?

If an end user runs go test all, that will also (necessarily) run the tests for runtime, so spurious failures in that package can add a lot of noise depending on the user's environment (compare #39204).

[prattmic]

This has only failed once, so failure rate doesn't currently seem too bad, but ultimately this is testing a best-effort feature that intentionally "breaks" after 5s, so it won't ever be 100% reliable (given a slow enough machine). That makes it a good candidate for #39204.

@zzkcode thanks for pointing out the total test time, I missed that. That does imply that 5s really did pass, rather than something else happening, like an early usleep return. It would be nice to get the core file to see what was happening. LUCI is theoretically capable of collecting extra files, but that isn't hooked up yet.

For now, perhaps we should modify the test to run a few times (3?) and at least verify that the functionality isn't failing 3/3 times?

[zzkcode]

Hi @prattmic, thanks for confirmation.

For now, perhaps we should modify the test to run a few times (3?) and at least verify that the functionality isn't failing 3/3 times?

IMHO, since this test case includes a small go+cgo program and only have few(5/6 maybe?) threads to quit, it should usually spend few seconds to exit. So we would expected to see this case pass(all threads received SIGQUIT and quit within 5 seconds) in the vast majority of cases, but not cover all cases(it will eventually quit anyway after 5 seconds). Like you said, however, it may be running on a slow enough machine(bullseye here, a kind of Raspberry Pi OS?), which is indeed slower comared to PCs.

I don't know whether this is a best choice to loop test more times, or just skip it on some platforms? I would prefer the former one for now since it only reported once as you mention before, but it won't supprise me if it continues fails 3 times in the future(rare event). So let's just loop 3 times, and if it pass, we just break and skip the rest loops. Otherwise, give it another 2 chances? It may cost more times(15s+) to run this single case in extreme scenarios. I want to submit a pr to fix this if this is OK.

One more tiny thing here(unrelated to this test, but related to the code change), I don't know if this is worth to be a change(maybe should fire another issue): as the 1.22rc1 is released, I had tried it to serveal times. It would usually spend at least 500ms(per loop) to quit when crash, kind of slower compared to 1.21 or earier releases(like sticking in somewhere for a very few moment while waiting the first thread to crash). Maybe we could use exponential backoff or just double the sleep time if this is a issue. Should not a big deal, just an experience thing.

FYI @cherrymui @bcmills

[bcmills]

In general it is not a good idea for tests to assume that something happens within any particular arbitrary timeframe. Is there some way to eliminate the arbitrary 5-second interval?

For example, could the test wait indefinitely instead of just for 5 seconds, or poll indefinitely to check whether the backtrace has been recorded yet? Then the timeout behavior of the test would follow the go test -timeout flag rather than an arbitrary machine-specific constant.