runtime/pprof: TestCPUProfileMultithreadMagnitude flaky due to Linux kernel bug

[prattmic]

2021-10-19T07:45:46-ee92daa/linux-386-longtest
2021-10-18T21:21:54-6c0daa7/linux-386-longtest
2021-10-18T19:46:27-394a1ad/linux-amd64-longtest
2021-10-15T20:29:48-85cbdda/linux-riscv64-jsing
2021-10-12T20:20:41-d032b2b/linux-amd64-wsl
2021-10-11T12:53:11-34f7b1f/linux-riscv64-unmatched
2021-10-08T17:32:31-d7ba1d2/linux-riscv64-unmatched
2021-10-08T14:08:12-5b9206f/linux-riscv64-unmatched
2021-10-06T16:11:20-aecf4b1/linux-riscv64-unmatched
2021-10-05T16:06:49-060cd73/linux-riscv64-unmatched
2021-10-02T13:52:59-07a7c6f/linux-riscv64-unmatched
2021-10-01T17:27:29-3357624/linux-riscv64-unmatched
2021-10-01T17:27:24-952df98/linux-riscv64-jsing
2021-10-01T04:30:22-822f349/linux-amd64-wsl
2021-09-30T16:43:32-9112d29/linux-riscv64-unmatched
2021-09-27T18:58:36-5b90958/linux-amd64-longtest
2021-09-27T18:58:36-5b90958/linux-amd64-wsl
2021-09-27T18:58:29-8cfd8c3/linux-amd64-longtest
2021-09-27T18:58:29-8cfd8c3/linux-amd64-wsl

Note that many of these include other profiling test failures which may indicate an issue with profiling beyond a fragile test.

cc @rhysh

[prattmic]

Some of the other tests I see failing in these, w.r.t. whether they were failing prior to the https://golang.org/cl/342054 patch stack (looking since April):

• TestCPUProfileInlining: 1 failure in August, and 1 in May, but failing for different reason.
• TestCPUProfileRecursion: no earlier failures.
• TestMathBigDivide: no earlier failures.
• TestMorestack: 4 failures in June on Darwin.
• TestCPUProfileLabel: no earlier failures.
• TestTimeVDSO: Added on Sept 22. Several earlier failures on Android, later marked as flaky on Android.

IMO, this is pretty strong evidence that these CLs have actually broke something w.r.t. profiling. I wouldn't be surprised if riscv or wsl have fundamental platform issues with profiling, but 386 and amd64 should certainly be OK.

[rhysh]

Thank you for the links. Here's what I've found so far in the 386 and amd64 (not WSL, not RISC-V) logs.

The failing test results for TestCPUProfileMultithreadMagnitude on those platforms all look like "got full profiling info, but only from a subset of threads", followed by some number of tests that get a very small number of profile samples—often 0, sometimes 1, sometimes dozens—when the tests expect several hundred. The samples that do appear come from the runtime, in the scheduler or scavenger, in functions that might run on otherwise-idle threads.

---

386

---

2021-10-19T07:45:46-ee92daa/linux-386-longtest

HOSTNAME=buildlet-linux-stretch-morecpu-rn873c60c, which per https://farmer.golang.org/builders is "Debian Stretch, but on e2-highcpu-16", which per https://cloud.google.com/compute/docs/general-purpose-machines#e2-high-cpu is 16 vCPUs and 16 GB memory.

    pprof_test.go:541: runtime/pprof.cpuHog1: 498
    pprof_test.go:541: runtime/pprof.cpuHog3: 3471
    --- FAIL: TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof (0.00s)
        pprof_test.go:179: compare 1m19.558967s vs 34.71s
        pprof_test.go:191: CPU usage reports are too different (limit -10.0%, got -56.4%)

The single-goroutine portion of TestCPUProfileMultithreadMagnitude saw 4.98 seconds of samples (in a 5-second test). The parallel portion (using GOMAXPROCS goroutines) also ran for 5 seconds, during which the OS reported 79.56 seconds of user+system time in rusage but the runtime only saw 34.71 seconds worth of samples. That looks like "the cliff at 2.5 that we got with setitimer is now present at 7 vCPU, sometimes".

The failure in that test run of TestCPUProfileMultithreadMagnitude is followed by failures in TestCPUProfileInlining, TestCPUProfileRecursion, TestMathBigDivide, TestMorestack, TestCPUProfileLabel, and TestTimeVDSO. All of those have "total 0 CPU profile samples collected" except for TestCPUProfileInlining which got a single sample (below the minimum of 125 and target of 500).

---

2021-10-18T21:21:54-6c0daa7/linux-386-longtest

HOSTNAME=buildlet-linux-stretch-morecpu-rnce27d47, so GOMAXPROCS=16 again.

    pprof_test.go:541: runtime/pprof.cpuHog1: 499
    pprof_test.go:541: runtime/pprof.cpuHog3: 2994
    --- FAIL: TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof (0.00s)
        pprof_test.go:179: compare 1m19.983449s vs 29.94s
        pprof_test.go:191: CPU usage reports are too different (limit -10.0%, got -62.6%)

The single-goroutine portion saw 4.99 seconds of samples, and the parallel portion saw 29.94 seconds of samples. This looks like "the cliff at 2.5 that we got with setitimer is now present at 6 vCPU, sometimes".

The failure in that test run of TestCPUProfileMultithreadMagnitude is followed by failures in TestCPUProfileRecursion, TestMathBigDivide, TestMorestack, TestCPUProfileLabel, and TestTimeVDSO. Again all of those failures were with "total 0 CPU profile samples collected" except for TestCPUProfileRecursion and TestMorestack which got a single sample each (vs min of 125 and target of 500).

---

amd64

---

2021-10-18T19:46:27-394a1ad/linux-amd64-longtest

No environment header at the top of this log.

    pprof_test.go:541: runtime/pprof.cpuHog1: 500
    pprof_test.go:541: runtime/pprof.cpuHog3: 4987
    --- FAIL: TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof (0.00s)
        pprof_test.go:179: compare 1m19.876446s vs 49.87s
        pprof_test.go:191: CPU usage reports are too different (limit -10.0%, got -37.6%)

The single-goroutine portion saw 5.00 seconds of samples, and the parallel portion saw 49.87 seconds of samples. This looks like "the cliff at 2.5 that we got with setitimer is now present at 10 vCPU, sometimes".

Followed by failure of TestMorestack (10 samples, 0 where expected), TestCPUProfileLabel (22 samples, 0 where expected), and TestTimeVDSO (9 samples, 0 where expected).

---

2021-09-27T18:58:36-5b90958/linux-amd64-longtest

HOSTNAME=buildlet-linux-stretch-morecpu-rn83cdb7c, so GOMAXPROCS=16.

    pprof_test.go:541: runtime/pprof.cpuHog1: 502
    pprof_test.go:541: runtime/pprof.cpuHog3: 5050
    --- FAIL: TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof (0.00s)
        pprof_test.go:179: compare 1m19.992473s vs 50.5s
        pprof_test.go:191: CPU usage reports are too different (limit -10.0%, got -36.9%)

The single-goroutine portion saw 5.02 seconds of samples (ran a little long!), and the parallel portion saw 50.50 seconds of samples (also ran a little long, but on fewer than GOMAXPROCS threads?). This could look like "the cliff at 2.5 that we got with setitimer is now present at 10 vCPU, sometimes" ... at least, 50.50 seconds is only slightly above 10 x 5.00 seconds.

There were no other failures in this run.

---

2021-09-27T18:58:29-8cfd8c3/linux-amd64-longtest

HOSTNAME=buildlet-linux-stretch-morecpu-rn6ff4f0f, so GOMAXPROCS=16.

    pprof_test.go:541: runtime/pprof.cpuHog1: 500
    pprof_test.go:541: runtime/pprof.cpuHog3: 5083
    --- FAIL: TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof (0.00s)
        pprof_test.go:179: compare 1m19.99259s vs 50.83s
        pprof_test.go:191: CPU usage reports are too different (limit -10.0%, got -36.5%)

The single-goroutine portion saw 5.00 seconds of samples, and the parallel portion saw 50.83 seconds of samples (ran a little long, but on fewer than GOMAXPROCS threads?). This could look like "the cliff at 2.5 that we got with setitimer is now present at 10 vCPU, sometimes" ... at least, 50.83 seconds is only slightly above 10 x 5.00 seconds.

There were no other failures in this run.

[gopherbot]

Change https://golang.org/cl/357650 mentions this issue: runtime: debug pprof test failures

[rhysh]

These failures look like three problems:

1. The failures on linux/amd64 for 8cfd8c3 and 5b90958 (both on 2021-09-27) are because I put the test too early in the CL chain. We discussed at runtime/pprof: Linux CPU profiles inaccurate beyond 250% CPU use #35057 (comment). The first WSL failures are on those two commits as well.
2. Failures in TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof. These appear on linux-riscv64, linux-amd64-wsl, and linux-ppc64. The first of these was 2021-09-30T16:43:32-9112d29, and they've occured regularly since.
3. Failures in TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof, coupled with failures in other CPU-profiling-related test cases. These appear on linux-amd64-longtest and linux-386-longtest. The first of these was 2021-10-18T19:46:27-394a1ad`

---

Problem 1 is a couple bad commits while the CL stack was landing. I'll filter those out of the results.

$ grep -r -l 'FAIL\truntime/pprof' . | egrep '2021-(09|10)' | grep '/linux-' | egrep '(8cfd8c3|5b90958)' | xargs grep -H 'FAIL:' | grep -v TestBlockProfile | sed -n -e 's,:[ ]*--- FAIL:,,p' | awk '{print $2,$1}' | sort | column -t
TestCPUProfileMultithreadMagnitude                                 ./2021-09-27T18:58:29-8cfd8c3/linux-amd64-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-09-27T18:58:29-8cfd8c3/linux-amd64-wsl
TestCPUProfileMultithreadMagnitude                                 ./2021-09-27T18:58:36-5b90958/linux-amd64-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-09-27T18:58:36-5b90958/linux-amd64-wsl
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-09-27T18:58:29-8cfd8c3/linux-amd64-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-09-27T18:58:29-8cfd8c3/linux-amd64-wsl
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-09-27T18:58:36-5b90958/linux-amd64-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-09-27T18:58:36-5b90958/linux-amd64-wsl

---

Problem 2 often shows garbage-collection-related functions in the profile. If the GC is running during the test, it would increase the CPU time for the whole process as reported by getrusage(2) without increasing the amount of time in the cpuHog functions that the test examines. I plan to update the test to disable the GC while it runs.

$ grep -r -l 'FAIL\truntime/pprof' . | egrep '2021-(09|10)' | grep '/linux-' | egrep -v '(8cfd8c3|5b90958)' | xargs grep -H 'FAIL:' | grep -v TestBlockProfile | sed -n -e 's,:[ ]*--- FAIL:,,p' | awk '{print $2,$1}' | sort | column -t | grep -v longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-09-30T16:43:32-9112d29/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude                                 ./2021-10-01T04:30:22-822f349/linux-amd64-wsl
TestCPUProfileMultithreadMagnitude                                 ./2021-10-01T17:27:24-952df98/linux-riscv64-jsing
TestCPUProfileMultithreadMagnitude                                 ./2021-10-01T17:27:29-3357624/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude                                 ./2021-10-02T13:52:59-07a7c6f/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude                                 ./2021-10-05T16:06:49-060cd73/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude                                 ./2021-10-06T16:11:20-aecf4b1/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude                                 ./2021-10-08T14:08:12-5b9206f/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude                                 ./2021-10-08T17:32:31-d7ba1d2/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude                                 ./2021-10-11T12:53:11-34f7b1f/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude                                 ./2021-10-12T20:20:41-d032b2b/linux-amd64-wsl
TestCPUProfileMultithreadMagnitude                                 ./2021-10-15T20:29:48-85cbdda/linux-riscv64-jsing
TestCPUProfileMultithreadMagnitude                                 ./2021-10-19T18:56:08-07e5527/linux-riscv64-unleashed
TestCPUProfileMultithreadMagnitude                                 ./2021-10-19T21:05:11-1b24c9e/linux-riscv64-unleashed
TestCPUProfileMultithreadMagnitude                                 ./2021-10-19T22:47:48-2be5b84/linux-riscv64-unleashed
TestCPUProfileMultithreadMagnitude                                 ./2021-10-20T05:28:45-b091189/linux-riscv64-unleashed
TestCPUProfileMultithreadMagnitude                                 ./2021-10-20T20:58:15-73652af/linux-ppc64-buildlet
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-09-30T16:43:32-9112d29/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-01T04:30:22-822f349/linux-amd64-wsl
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-01T17:27:24-952df98/linux-riscv64-jsing
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-01T17:27:29-3357624/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-02T13:52:59-07a7c6f/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-05T16:06:49-060cd73/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-06T16:11:20-aecf4b1/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-08T14:08:12-5b9206f/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-08T17:32:31-d7ba1d2/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-11T12:53:11-34f7b1f/linux-riscv64-unmatched
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-12T20:20:41-d032b2b/linux-amd64-wsl
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-15T20:29:48-85cbdda/linux-riscv64-jsing
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-19T18:56:08-07e5527/linux-riscv64-unleashed
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-19T21:05:11-1b24c9e/linux-riscv64-unleashed
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-19T22:47:48-2be5b84/linux-riscv64-unleashed
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-20T05:28:45-b091189/linux-riscv64-unleashed
TestCPUProfileMultithreadMagnitude/serial_execution_OS_vs_pprof    ./2021-10-20T20:58:15-73652af/linux-ppc64-buildlet

---

Problem 3 is new in the last few days. When it shows up, it affects the single-goroutine CPU profiling tests too. It's only appeared on the longtest builders. Those have 16 hyperthreads / GOMAXPROCS=16.

The failures in single-goroutine tests take the form of zero or near-zero samples in the desired functions, and the failures in TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof take the form of the CPU profile reporting only 6 to 14 threads worth of CPU time. (The tests expects 16 threads worth of reporting, and its 10% error margin means that reports from 15 threads worth of work would be considered a success.)

It's interesting that the amount of CPU time reported in the profile is always very close to an integer number of threads (each running for 5 seconds). It's also interesting that the smallest report is 6 out of the 16 threads. I disabled the new timer_create-based profiler and ran the test on the linux-386-longtest builder, and found that setitimer is able to deliver enough SIGPROF events to the process to cover the work of 10 threads (https://storage.googleapis.com/go-build-log/d3ff14b2/linux-386-longtest_a3d8929c.log). My working theory on this failure mode is that on some fraction of the process's threads, the kernel delivers the signals from setitimer first, so the deliveries from timer_create are crowded out up to setitimer's maximum signal delivery rate.

I also find it suspicious that this failure mode is new in the last few days, but none of the nearby commits look related and I also haven't been able to reproduce the problem myself.

$ grep -r -l 'FAIL\truntime/pprof' . | egrep '2021-(09|10)' | grep '/linux-' | egrep -v '(8cfd8c3|5b90958)' | xargs grep -H 'FAIL:' | grep -v TestBlockProfile | sed -n -e 's,:[ ]*--- FAIL:,,p' | awk '{print $2,$1}' | sort | column -t | grep longtest
TestCPUProfileInlining                                             ./2021-10-19T07:45:46-ee92daa/linux-386-longtest
TestCPUProfileInlining                                             ./2021-10-19T11:52:34-bde0463/linux-386-longtest
TestCPUProfileInlining                                             ./2021-10-19T18:42:57-7999fd4/linux-386-longtest
TestCPUProfileInlining                                             ./2021-10-19T22:47:48-2be5b84/linux-386-longtest
TestCPUProfileInlining                                             ./2021-10-20T07:49:28-d2ec73c/linux-amd64-longtest
TestCPUProfileInlining                                             ./2021-10-20T19:29:12-4320949/linux-386-longtest
TestCPUProfileInlining                                             ./2021-10-20T20:58:15-73652af/linux-386-longtest
TestCPUProfileLabel                                                ./2021-10-18T19:46:27-394a1ad/linux-amd64-longtest
TestCPUProfileLabel                                                ./2021-10-18T21:21:54-6c0daa7/linux-386-longtest
TestCPUProfileLabel                                                ./2021-10-19T07:45:46-ee92daa/linux-386-longtest
TestCPUProfileLabel                                                ./2021-10-19T11:52:34-bde0463/linux-386-longtest
TestCPUProfileLabel                                                ./2021-10-19T22:47:48-2be5b84/linux-386-longtest
TestCPUProfileLabel                                                ./2021-10-20T00:05:22-4cc6a91/linux-386-longtest
TestCPUProfileLabel                                                ./2021-10-20T20:39:19-d419a80/linux-386-longtest
TestCPUProfileLabel                                                ./2021-10-20T20:58:15-73652af/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-18T19:46:27-394a1ad/linux-amd64-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-18T21:21:54-6c0daa7/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-19T07:45:46-ee92daa/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-19T11:52:34-bde0463/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-19T16:19:27-f92a358/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-19T17:01:35-254c497/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-19T18:42:57-7999fd4/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-19T22:47:48-2be5b84/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-20T00:05:22-4cc6a91/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-20T07:49:28-d2ec73c/linux-amd64-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-20T19:29:12-4320949/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-20T20:39:19-d419a80/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-20T20:39:25-e904928/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-20T20:58:15-73652af/linux-386-longtest
TestCPUProfileMultithreadMagnitude                                 ./2021-10-21T15:42:41-8b0bea9/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-18T19:46:27-394a1ad/linux-amd64-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-18T21:21:54-6c0daa7/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-19T07:45:46-ee92daa/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-19T11:52:34-bde0463/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-19T16:19:27-f92a358/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-19T17:01:35-254c497/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-19T18:42:57-7999fd4/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-19T22:47:48-2be5b84/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-20T00:05:22-4cc6a91/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-20T07:49:28-d2ec73c/linux-amd64-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-20T19:29:12-4320949/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-20T20:39:19-d419a80/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-20T20:39:25-e904928/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-20T20:58:15-73652af/linux-386-longtest
TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof  ./2021-10-21T15:42:41-8b0bea9/linux-386-longtest
TestCPUProfileRecursion                                            ./2021-10-18T21:21:54-6c0daa7/linux-386-longtest
TestCPUProfileRecursion                                            ./2021-10-19T07:45:46-ee92daa/linux-386-longtest
TestCPUProfileRecursion                                            ./2021-10-19T11:52:34-bde0463/linux-386-longtest
TestCPUProfileRecursion                                            ./2021-10-19T22:47:48-2be5b84/linux-386-longtest
TestCPUProfileRecursion                                            ./2021-10-20T00:05:22-4cc6a91/linux-386-longtest
TestCPUProfileRecursion                                            ./2021-10-20T20:58:15-73652af/linux-386-longtest
TestMathBigDivide                                                  ./2021-10-18T21:21:54-6c0daa7/linux-386-longtest
TestMathBigDivide                                                  ./2021-10-19T07:45:46-ee92daa/linux-386-longtest
TestMathBigDivide                                                  ./2021-10-19T16:19:27-f92a358/linux-386-longtest
TestMathBigDivide                                                  ./2021-10-20T20:58:15-73652af/linux-386-longtest
TestMorestack                                                      ./2021-10-18T19:46:27-394a1ad/linux-amd64-longtest
TestMorestack                                                      ./2021-10-18T21:21:54-6c0daa7/linux-386-longtest
TestMorestack                                                      ./2021-10-19T07:45:46-ee92daa/linux-386-longtest
TestMorestack                                                      ./2021-10-19T11:52:34-bde0463/linux-386-longtest
TestMorestack                                                      ./2021-10-20T20:39:19-d419a80/linux-386-longtest
TestTimeVDSO                                                       ./2021-10-18T19:46:27-394a1ad/linux-amd64-longtest
TestTimeVDSO                                                       ./2021-10-18T21:21:54-6c0daa7/linux-386-longtest
TestTimeVDSO                                                       ./2021-10-19T07:45:46-ee92daa/linux-386-longtest
TestTimeVDSO                                                       ./2021-10-19T11:52:34-bde0463/linux-386-longtest
TestTimeVDSO                                                       ./2021-10-19T16:19:27-f92a358/linux-386-longtest
TestTimeVDSO                                                       ./2021-10-20T00:05:22-4cc6a91/linux-386-longtest
TestTimeVDSO                                                       ./2021-10-21T15:42:41-8b0bea9/linux-386-longtest

$ find . -name linux-386-longtest | egrep -v '(8cfd8c3|5b90958)' | egrep '2021-(09|10)' | sort | xargs egrep 'pprof_test.go.*compare'
./2021-10-18T21:21:54-6c0daa7/linux-386-longtest:        pprof_test.go:179: compare 1m19.983449s vs 29.94s
./2021-10-19T07:45:46-ee92daa/linux-386-longtest:        pprof_test.go:179: compare 1m19.558967s vs 34.71s
./2021-10-19T11:52:34-bde0463/linux-386-longtest:        pprof_test.go:179: compare 1m19.932465s vs 34.89s
./2021-10-19T16:19:27-f92a358/linux-386-longtest:        pprof_test.go:179: compare 1m19.987855s vs 59.95s
./2021-10-19T17:01:35-254c497/linux-386-longtest:        pprof_test.go:179: compare 1m19.980093s vs 1m4.9s
./2021-10-19T18:42:57-7999fd4/linux-386-longtest:        pprof_test.go:179: compare 1m19.950427s vs 1m4.86s
./2021-10-19T22:47:48-2be5b84/linux-386-longtest:        pprof_test.go:179: compare 1m19.920228s vs 1m4.86s
./2021-10-20T00:05:22-4cc6a91/linux-386-longtest:        pprof_test.go:179: compare 1m19.976284s vs 54.89s
./2021-10-20T19:29:12-4320949/linux-386-longtest:        pprof_test.go:179: compare 1m19.86348s vs 59.83s
./2021-10-20T20:39:19-d419a80/linux-386-longtest:        pprof_test.go:179: compare 1m19.975459s vs 1m9.87s
./2021-10-20T20:39:25-e904928/linux-386-longtest:        pprof_test.go:179: compare 1m18.992738s vs 44.33s
./2021-10-20T20:58:15-73652af/linux-386-longtest:        pprof_test.go:179: compare 1m19.982008s vs 1m4.89s
./2021-10-21T15:42:41-8b0bea9/linux-386-longtest:        pprof_test.go:179: compare 1m19.972142s vs 1m4.91s

[rhysh]

I got an interesting trybot failure on https://go-review.googlesource.com/c/go/+/357650/8: https://storage.googleapis.com/go-build-log/c3d0e67e/linux-386-longtest_222a0fb6.log

The result in TestCPUProfileMultithreadMagnitude/parallel_execution_OS_vs_pprof shows that the profile has samples for 1m14.77s, versus the 1m19.882056s of time reported by getrusage(2). (That would usually be within the acceptable bounds for the long-form test, but I've changed it to always fail and dump out the results for inspection.) That test ran with GOMAXPROCS=16, and the number of samples lines up with only 15 of those 16 threads being profiled. The labels in the profile confirm: there are no samples from worker:parallel-7. The parallel portion of the test (for the moment, in that debugging CL) uses runtime.LockOSThread.

It would be more interesting if the test run showed more than a single thread not reporting in at all, but this is still pretty good.

$ pbpaste | grep worker | awk '{print $2,$NF,$1}' | sort -k 2
0x8171ffc {worker:parallel-0} 109:
0x80ab580 {worker:parallel-0} 1:
0x8171fff {worker:parallel-0} 24:
0x8172002 {worker:parallel-0} 365:
0x8171ffc {worker:parallel-10} 122:
0x8171fff {worker:parallel-10} 16:
0x8171fea {worker:parallel-10} 2:
0x8172002 {worker:parallel-10} 360:
0x8171ffc {worker:parallel-11} 123:
0x8171fff {worker:parallel-11} 29:
0x8172002 {worker:parallel-11} 346:
0x8171ffc {worker:parallel-12} 107:
0x8171fff {worker:parallel-12} 21:
0x8172002 {worker:parallel-12} 372:
0x8171ffc {worker:parallel-13} 123:
0x8171fff {worker:parallel-13} 12:
0x8171fea {worker:parallel-13} 1:
0x8172002 {worker:parallel-13} 363:
0x8171ffc {worker:parallel-14} 124:
0x80ab580 {worker:parallel-14} 1:
0x80bc65d {worker:parallel-14} 1:
0x8171fff {worker:parallel-14} 24:
0x8172002 {worker:parallel-14} 349:
0x8171ffc {worker:parallel-15} 108:
0x8171fff {worker:parallel-15} 18:
0x80ab1e9 {worker:parallel-15} 1:
0x8171fea {worker:parallel-15} 1:
0x8172002 {worker:parallel-15} 362:
0x80ab580 {worker:parallel-15} 3:
0x80ab580 {worker:parallel-15} 3:
0x8171ffc {worker:parallel-1} 110:
0x80ab580 {worker:parallel-1} 1:
0x80ab580 {worker:parallel-1} 1:
0x8171fea {worker:parallel-1} 1:
0x8171fff {worker:parallel-1} 25:
0x8172002 {worker:parallel-1} 362:
0x8171ffc {worker:parallel-2} 137:
0x8171fff {worker:parallel-2} 23:
0x8172002 {worker:parallel-2} 339:
0x8171ffc {worker:parallel-3} 133:
0x8171fff {worker:parallel-3} 21:
0x8172002 {worker:parallel-3} 344:
0x8171ffc {worker:parallel-4} 125:
0x8171fff {worker:parallel-4} 17:
0x8172002 {worker:parallel-4} 356:
0x8171ffc {worker:parallel-5} 128:
0x8171fff {worker:parallel-5} 16:
0x8172002 {worker:parallel-5} 354:
0x8171ffc {worker:parallel-6} 126:
0x8171f09 {worker:parallel-6} 1:
0x8171fff {worker:parallel-6} 27:
0x8172002 {worker:parallel-6} 344:
0x8171ffc {worker:parallel-8} 128:
0x8171fff {worker:parallel-8} 17:
0x8172002 {worker:parallel-8} 353:
0x8171ffc {worker:parallel-9} 124:
0x8171fff {worker:parallel-9} 23:
0x8172002 {worker:parallel-9} 353:
0x8171f7c {worker:serial} 118:
0x80ab580 {worker:serial} 1:
0x80ab580 {worker:serial} 1:
0x8171f7f {worker:serial} 2:
0x8171f82 {worker:serial} 377:

[rhysh]

I'd like to see the output of perf trace --no-syscalls -e 'signal:*' when the parallel portion of the test fails (as seen on linux-{386,amd64}-longtest), but I haven't been able to reproduce the failure on a system I control.

Gomote access might help, but the instructions at https://github.com/golang/go/wiki/Gomote#access-token say that new accounts are on hold.

I've built the toolchain with GOARCH=386 and GOHOSTARCH=386 on a 16-hyperthread machine (a c5.4xlarge Intel EC2 instance) running Debian Stretch and have run go test runtime/pprof -test.count=1 many times (198 so far, versus the 16 failures of linux-386-longtest across 66 recent commits), but have not seen a single failure in that package's tests on my test machine.

In the linux-386-longtest failures, the first failing test is TestCPUProfileMultithreadMagnitude. The part of that test that fails is the one that exercises GOMAXPROCS threads/goroutines in parallel. If that test fails, subsequent tests of the CPU profiler may also fail, including ones that use a single thread/goroutine. However, 1/ the two tests of the CPU profiler preceding TestCPUProfileMultithreadMagnitude (TestCPUProfile using 1 thread/goroutine and TestCPUProfileMultithreaded using 2 threads/goroutines) do not fail, and 2/ the single-threaded / serial portion (the first half) of TestCPUProfileMultithreadMagnitude also does not fail.

It looks like CPU profiling is working for the process, then the process uses lots of threads in parallel, and then CPU profiling no longer consistently works for the process. And, this is new as of October 18.

I don't know how to move forward with the "longtest" half of the problem. Do you have advice, @prattmic ?

• Are gomote access requests being processed, and am I the sort of person (have "may-start-trybots", haven't pursued gerrit "approvers") who would get access? I see recent requests being denied.
• Is there someone with gomote access who is able to reproduce the failure, and could share the perf-trace results (plus the test log output) from a failing run?
• Is there something I'm missing in my repro attempts—especially something that lets me stay on EC2, where I have easy access through my employer?

My rough non-repro steps:

% ssh admin@vm-ip
Linux ip-172-31-53-15 4.9.0-16-amd64 #1 SMP Debian 4.9.272-2 (2021-07-19) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

$ sudo apt-get install git build-essential libc6-dev-i386
$ git clone https://go.googlesource.com/go ~/go-bisect
$ git -C ~/go-bisect worktree add --detach ~/go1.4
$ git -C ~/go1.4 checkout origin/release-branch.go1.4
$ (cd ~/go1.4/src && ./make.bash)
$ git -C ~/go-bisect checkout 6c0daa733192031eab23d09ed6515c4cd959aa92
$ (cd ~/go-bisect/src && env "GOARCH=386" "GOHOSTARCH=386" ./make.bash)
$ ~/go-bisect/bin/go test runtime/pprof -test.count=1
$ # Though in practice I copied the stress tool to the machine and am running 
$ # $ nohup ./stress.linux_amd64.exe -p 1 ~/go-bisect/bin/go test runtime/pprof -test.count=1 &

[bcmills]

The failure rate of runtime/pprof has become quite high on the linux-386-longtest builder since around Oct. 18.

That builder is especially important for this release cycle, because it is the only longtest builder running non-short fuzz tests with non-instrumented binaries and has been catching a lot of fuzzing testing issues. We need to do something to get it to produce useful coverage again — perhaps either rolling back runtime/pprof changes, or adding testenv.SkipFlaky calls to specific new tests that are failing. (CC @golang/release)

greplogs --dashboard -md -l -e FAIL\\s+runtime/pprof --since=2021-09-01

Details

2021-10-25T16:07:11-c6e82e5/linux-amd64-longtest
2021-10-25T03:00:02-7b55457/linux-386-longtest
2021-10-25T02:59:35-f686f6a/linux-amd64-wsl
2021-10-23T20:23:35-8dbf3e9/linux-386-longtest
2021-10-23T16:18:19-16318df/linux-386-longtest
2021-10-23T16:18:19-16318df/linux-amd64-longtest
2021-10-23T16:18:15-12e4404/linux-amd64-longtest
2021-10-23T16:17:58-3cd28ba/linux-386-longtest
2021-10-22T21:12:07-76dd01f/linux-386-longtest
2021-10-21T18:20:19-f99e40a/linux-386-longtest
2021-10-21T17:31:42-392bb06/linux-386-longtest
2021-10-21T15:45:05-6ec9a1d/linux-386-longtest
2021-10-21T15:42:41-8b0bea9/linux-386-longtest
2021-10-20T20:58:15-73652af/linux-386-longtest
2021-10-20T20:58:15-73652af/linux-ppc64-buildlet
2021-10-20T20:39:25-e904928/linux-386-longtest
2021-10-20T20:39:19-d419a80/linux-386-longtest
2021-10-20T19:29:12-4320949/linux-386-longtest
2021-10-20T07:49:28-d2ec73c/linux-amd64-longtest
2021-10-20T05:28:45-b091189/linux-riscv64-unleashed
2021-10-20T00:05:22-4cc6a91/linux-386-longtest
2021-10-19T22:47:48-2be5b84/linux-386-longtest
2021-10-19T22:47:48-2be5b84/linux-riscv64-unleashed
2021-10-19T21:05:11-1b24c9e/linux-riscv64-unleashed
2021-10-19T18:56:08-07e5527/linux-riscv64-unleashed
2021-10-19T18:42:57-7999fd4/linux-386-longtest
2021-10-19T17:01:35-254c497/linux-386-longtest
2021-10-19T16:19:27-f92a358/linux-386-longtest
2021-10-19T11:52:34-bde0463/linux-386-longtest
2021-10-19T07:45:46-ee92daa/linux-386-longtest
2021-10-18T21:21:54-6c0daa7/linux-386-longtest
2021-10-18T19:46:27-394a1ad/linux-amd64-longtest
2021-10-15T20:29:48-85cbdda/linux-riscv64-jsing
2021-10-12T20:20:41-d032b2b/linux-amd64-wsl
2021-10-11T20:46:14-7023535/linux-ppc64-buildlet
2021-10-11T12:53:11-34f7b1f/linux-riscv64-unmatched
2021-10-08T17:32:31-d7ba1d2/linux-riscv64-unmatched
2021-10-08T14:08:12-5b9206f/linux-riscv64-unmatched
2021-10-08T02:09:58-e74db46/openbsd-arm64-jsing
2021-10-06T16:11:20-aecf4b1/linux-riscv64-unmatched
2021-10-05T16:06:49-060cd73/linux-riscv64-unmatched
2021-10-02T13:52:59-07a7c6f/linux-riscv64-unmatched
2021-10-02T00:44:24-afe43f1/linux-riscv64-unmatched
2021-10-02T00:44:24-afe43f1/openbsd-arm64-jsing
2021-10-01T17:27:29-3357624/linux-riscv64-unmatched
2021-10-01T17:27:24-952df98/linux-riscv64-jsing
2021-10-01T04:30:22-822f349/linux-amd64-wsl
2021-09-30T16:43:32-9112d29/linux-riscv64-unmatched
2021-09-27T21:31:14-04f7521/android-amd64-emu
2021-09-27T20:58:06-e54843f/android-arm-corellium
2021-09-27T18:58:36-5b90958/android-arm-corellium
2021-09-27T18:58:36-5b90958/linux-amd64-longtest
2021-09-27T18:58:36-5b90958/linux-amd64-wsl
2021-09-27T18:58:29-8cfd8c3/linux-amd64-longtest
2021-09-27T18:58:29-8cfd8c3/linux-amd64-wsl
2021-09-27T18:57:20-3d795ea/android-amd64-emu
2021-09-27T18:14:10-ecac351/android-arm-corellium
2021-09-27T16:16:57-d5cfba0/android-arm-corellium
2021-09-27T11:52:07-6e5dd0b/android-arm-corellium
2021-09-27T10:10:16-dac89a9/android-arm-corellium
2021-09-27T05:10:56-54079df/android-amd64-emu
2021-09-27T03:07:49-ff8a7e5/android-arm-corellium
2021-09-24T21:42:31-d60ad1e/android-amd64-emu
2021-09-24T21:38:56-1ce6fd0/android-arm-corellium
2021-09-24T19:39:49-c94543b/android-arm-corellium
2021-09-24T18:21:14-f6b5ffb/android-arm-corellium
2021-09-24T17:53:04-b00222f/android-arm-corellium
2021-09-24T17:52:54-584afc2/android-arm-corellium
2021-09-24T17:23:19-d5d3f80/android-arm-corellium
2021-09-24T14:52:47-217507e/android-amd64-emu
2021-09-24T14:52:47-217507e/android-arm-corellium
2021-09-24T01:25:24-483533d/openbsd-arm64-jsing

[gopherbot]

Change https://golang.org/cl/358574 mentions this issue: runtime: disable timer_create profiler on 386

[rhysh]

I think the longtest failures are related to the Linux Kernel version in use.

From one side, the linux-386-longtest failures started abruptly on 2021-10-18 and none of the Go commits close to the first failure look like they'd be related. It looks like the linux-386-longtest runs as a container on GCE and does not require nested virtualization, and so runs on the latest stable version of GCE's Container Optimized OS.

https://github.com/golang/build/blob/a72dd9f27a813c2d30db1b6d0f058f0fe2b32751/dashboard/builders.go#L1830
https://github.com/golang/build/blob/a72dd9f27a813c2d30db1b6d0f058f0fe2b32751/buildlet/gce.go#L84
https://github.com/golang/build/blob/a72dd9f27a813c2d30db1b6d0f058f0fe2b32751/buildlet/gce.go#L346

Following on to that, it looks like COS m93 with Kernel 5.10.68 became a stable release on 2021-10-18, replacing m87 with Kernel 5.4.144. (I only have a date, not a timestamp ... I haven't figured out how to list VM images without a billing-enabled account.) https://cloud.google.com/container-optimized-os/docs/release-notes/m93 https://cloud.google.com/container-optimized-os/docs/release-notes/m89

From the other side, I've seen the following on 16-thread machines I control:

Debian Stretch
Successes: 4270
Failures: 0
uname -a: Linux ip-172-31-53-15 4.9.0-16-amd64 #1 SMP Debian 4.9.272-2 (2021-07-19) x86_64 GNU/Linux

Ubuntu 21.04
Successes: 75
Failures: 1
uname -a: Linux ip-172-31-57-140 5.11.0-1007-aws #7-Ubuntu SMP Thu May 6 20:17:21 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Ubuntu 21.10
Successes: 47
Failures: 0
uname -a: Linux ip-172-31-51-72 5.13.0-1005-aws #6-Ubuntu SMP Fri Oct 8 07:48:02 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I've only gotten one failure so far, and I don't have the perf trace output from it, but it's a start.

@golang/release , does the "kernel / COS image version might be related to these failures" idea sound plausible to you? Did the container-based builders pick up COS m93, and does the timeline of that match this set of linux-386-longtest failures? (I don't know if I'm reading and interpreting the code correctly.) Thanks.

[rhysh]

I continue to see about a 1% failure rate on my test machines that use very new kernels.

Debian Stretch, Linux 4.9: 4492 runs, 0 failures
Ubuntu 21.04, Linux 5.11: 302 runs, 5 failures
Ubuntu 21.10, Linux 5.13: 271 runs, 3 failures

They're c5.4xlarge machines in us-west-2, running images ami-025f3bcb64ebe6e83, ami-02684f1c7f36cdd5b, ami-0d4a468c8fcc4b5f0.

I've started an additional Ubuntu 21.10 machine to collect perf trace output.

$ sudo apt-get install linux-tools-common linux-tools-aws
$ ~/go-bisect/bin/go test -c -o ~/pprof.test runtime/pprof
$ (cd /home/ubuntu/go-bisect/src/runtime/pprof/ && \
sudo nohup /home/ubuntu/stress.linux_amd64.exe -p=1 -failure='FAIL' \
bash -c 'perf trace --no-syscalls -e '\''signal:*'\'' /home/ubuntu/pprof.test -test.count=1 ; false' \
&>>/home/ubuntu/nohup.out &)