runtime: TestRuntimeLockMetricsAndProfile failures

[gopherbot]

#!watchflakes
post <- pkg == "runtime" && test ~ `TestRuntimeLockMetricsAndProfile`

Issue created automatically to collect these failures.

Example (log):

--- FAIL: TestRuntimeLockMetricsAndProfile (0.03s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock (0.02s)
        --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock/sample-1 (0.01s)
            metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.006333s)
            metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.006331s)
            metrics_test.go:1077: stack [runtime.unlock runtime_test.TestRuntimeLockMetricsAndProfile.func5.1 runtime_test.(*contentionWorker).run] has samples totaling n=99 value=6332772
            metrics_test.go:1156: mutex profile reported contention count different from the known true count (99 != 100)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "runtime" && test == "TestRuntimeLockMetricsAndProfile"

2023-11-17 23:04 linux-ppc64-sid-power10 go@0b31a46f runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.03s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock (0.02s)
        --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock/sample-1 (0.01s)
            metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.006333s)
            metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.006331s)
            metrics_test.go:1077: stack [runtime.unlock runtime_test.TestRuntimeLockMetricsAndProfile.func5.1 runtime_test.(*contentionWorker).run] has samples totaling n=99 value=6332772
            metrics_test.go:1156: mutex profile reported contention count different from the known true count (99 != 100)

2023-11-17 23:04 windows-386-2016 go@0b31a46f runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.21s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.semrelease (0.01s)
        metrics_test.go:1041: no increase in /sync/mutex/wait/total:seconds metric
        metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.000335s)
        metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.000000s)
        metrics_test.go:1077: stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run] has samples totaling n=491 value=173075

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "runtime" && test == "TestRuntimeLockMetricsAndProfile"

2023-11-17 23:16 windows-amd64-longtest go@f664031b runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.22s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.semrelease (0.02s)
        metrics_test.go:1041: no increase in /sync/mutex/wait/total:seconds metric
        metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.002332s)
        metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.000000s)
        metrics_test.go:1077: stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run] has samples totaling n=3062 value=988506

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "runtime" && test == "TestRuntimeLockMetricsAndProfile"

2023-11-17 23:16 linux-ppc64-sid-power10 go@f664031b runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.03s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock (0.02s)
        --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock/sample-1 (0.01s)
            metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.006328s)
            metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.006326s)
            metrics_test.go:1077: stack [runtime.unlock runtime_test.TestRuntimeLockMetricsAndProfile.func5.1 runtime_test.(*contentionWorker).run] has samples totaling n=99 value=6328117
            metrics_test.go:1156: mutex profile reported contention count different from the known true count (99 != 100)

2023-11-17 23:34 linux-ppc64-sid-power10 go@631a6c2a runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.02s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock (0.01s)
        --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock/sample-1 (0.01s)
            metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.005366s)
            metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.005365s)
            metrics_test.go:1077: stack [runtime.unlock runtime_test.TestRuntimeLockMetricsAndProfile.func5.1 runtime_test.(*contentionWorker).run] has samples totaling n=86 value=5366454
            metrics_test.go:1156: mutex profile reported contention count different from the known true count (86 != 100)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "runtime" && test == "TestRuntimeLockMetricsAndProfile"

2023-11-17 23:15 windows-arm64-11 go@f67b2d8f runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.16s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.semrelease (0.01s)
        metrics_test.go:1041: no increase in /sync/mutex/wait/total:seconds metric
        metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.003580s)
        metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.000000s)
        metrics_test.go:1077: stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run] has samples totaling n=3101 value=2156101

2023-11-17 23:16 windows-arm64-11 go@f664031b runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.13s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.semrelease (0.02s)
        metrics_test.go:1041: no increase in /sync/mutex/wait/total:seconds metric
        metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.003674s)
        metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.000000s)
        metrics_test.go:1077: stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run] has samples totaling n=3308 value=2331713

2023-11-17 23:34 windows-arm64-11 go@631a6c2a runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.21s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.semrelease (0.07s)
        metrics_test.go:1041: no increase in /sync/mutex/wait/total:seconds metric
        metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.005268s)
        metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.000000s)
        metrics_test.go:1077: stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run] has samples totaling n=5951 value=3491388

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "runtime" && test == "TestRuntimeLockMetricsAndProfile"

2023-11-19 02:15 windows-amd64-2016 go@aa9dd500 runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.22s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.semrelease (0.02s)
        metrics_test.go:1041: no increase in /sync/mutex/wait/total:seconds metric
        metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.001346s)
        metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.000000s)
        metrics_test.go:1077: stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run] has samples totaling n=3436 value=1006621

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "runtime" && test == "TestRuntimeLockMetricsAndProfile"

2023-11-19 07:31 linux-ppc64-sid-power10 go@d67ac938 runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.02s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock (0.02s)
        --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock/sample-1 (0.01s)
            metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.006294s)
            metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.006293s)
            metrics_test.go:1077: stack [runtime.unlock runtime_test.TestRuntimeLockMetricsAndProfile.func5.1 runtime_test.(*contentionWorker).run] has samples totaling n=99 value=6294492
            metrics_test.go:1156: mutex profile reported contention count different from the known true count (99 != 100)

2023-11-19 07:31 windows-arm64-11 go@d67ac938 runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.16s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.semrelease (0.05s)
        metrics_test.go:1041: no increase in /sync/mutex/wait/total:seconds metric
        metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.009316s)
        metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.000000s)
        metrics_test.go:1077: stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run] has samples totaling n=8900 value=6224493

2023-11-19 15:24 linux-ppc64-sid-power10 go@1c15291f runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.02s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock (0.01s)
        --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock/sample-1 (0.01s)
            metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.006170s)
            metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.006168s)
            metrics_test.go:1077: stack [runtime.unlock runtime_test.TestRuntimeLockMetricsAndProfile.func5.1 runtime_test.(*contentionWorker).run] has samples totaling n=99 value=6169552
            metrics_test.go:1156: mutex profile reported contention count different from the known true count (99 != 100)

2023-11-19 15:24 windows-arm64-11 go@1c15291f runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.16s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.semrelease (0.05s)
        metrics_test.go:1041: no increase in /sync/mutex/wait/total:seconds metric
        metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.006365s)
        metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.000000s)
        metrics_test.go:1077: stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run] has samples totaling n=4658 value=3628090

2023-11-19 17:06 linux-ppc64-sid-power10 go@2551fffd runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.01s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock (0.01s)
        --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock/sample-1 (0.01s)
            metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.000038s)
            metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.000038s)
            metrics_test.go:1077: stack [runtime.unlock runtime_test.TestRuntimeLockMetricsAndProfile.func5.1 runtime_test.(*contentionWorker).run] has samples totaling n=2 value=37499
            metrics_test.go:1153: runtime/metrics reported less than the known minimum contention duration (0.000038s < 0.001000s)
            metrics_test.go:1156: mutex profile reported contention count different from the known true count (2 != 100)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "runtime" && test == "TestRuntimeLockMetricsAndProfile"

2023-11-19 21:11 windows-arm64-11 go@06145fe0 runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.15s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.semrelease (0.01s)
        metrics_test.go:1041: no increase in /sync/mutex/wait/total:seconds metric
        metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.002936s)
        metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.000000s)
        metrics_test.go:1077: stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run] has samples totaling n=2560 value=1676526

2023-11-19 22:05 windows-arm64-11 go@63828938 runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.23s)
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.semrelease (0.08s)
        metrics_test.go:1041: no increase in /sync/mutex/wait/total:seconds metric
        metrics_test.go:1048: lock contention growth in runtime/pprof's view  (0.007995s)
        metrics_test.go:1049: lock contention growth in runtime/metrics' view (0.000000s)
        metrics_test.go:1077: stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run] has samples totaling n=5920 value=5630416

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "runtime" && test == "TestRuntimeLockMetricsAndProfile"

2023-11-21 21:30 linux-ppc64le-buildlet go@41f58b22 runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.10s)
    metrics_test.go:967: NumCPU 4
    metrics_test.go:968: GOMAXPROCS 4
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock (0.08s)
        --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.lock/sample-2 (0.05s)
            metrics_test.go:1056: lock contention growth in runtime/pprof's view  (0.020478s)
            metrics_test.go:1057: lock contention growth in runtime/metrics' view (0.081851s)
            metrics_test.go:1085: stack [runtime.unlock runtime_test.TestRuntimeLockMetricsAndProfile.func5.1 runtime_test.(*contentionWorker).run] has samples totaling n=106 value=20478295
            metrics_test.go:1199: views differ by more than 3.750000x

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "runtime" && test == "TestRuntimeLockMetricsAndProfile"

2023-11-21 21:27 linux-386-longtest go@48a6362d runtime.TestRuntimeLockMetricsAndProfile (log)

--- FAIL: TestRuntimeLockMetricsAndProfile (0.17s)
    metrics_test.go:967: NumCPU 16
    metrics_test.go:968: GOMAXPROCS 16
    --- FAIL: TestRuntimeLockMetricsAndProfile/runtime.semrelease (0.05s)
        metrics_test.go:1056: lock contention growth in runtime/pprof's view  (0.000048s)
        metrics_test.go:1057: lock contention growth in runtime/metrics' view (0.000049s)
        metrics_test.go:1085: stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run] has samples totaling n=0 value=0
        metrics_test.go:1087: profile:
            PeriodType: contentions count
            Period: 1
...
            7: 0x8048000/0x841a000/0x0 /workdir/tmp/go-build1114361604/b001/runtime.test  [FN]
            8: 0xf7d10000/0xf7e64000/0x19000 /lib32/libc-2.31.so 1a7ffd46bcc790ed7017919b4ca6fc812c8c615c 
            9: 0xf7ee4000/0xf7ef5000/0x5000 /lib32/libpthread-2.31.so 505f3eecd0c7c22229602b598ee187d5b5a94bbb 
            10: 0xf7f04000/0xf7f11000/0x3000 /lib32/libresolv-2.31.so d8095c858bdd3a4d2a534300f09990a6a0644ea4 
            11: 0xf7f28000/0xf7f2a000/0x0 [vdso]  
            12: 0xf7f2b000/0xf7f48000/0x1000 /lib32/ld-2.31.so dd55267740b58ae626dc11439ff85e923974f7d5 
        metrics_test.go:1089: have stack [runtime.unlock runtime_test.TestRuntimeLockMetricsAndProfile.func5.1 runtime_test.(*contentionWorker).run]
        metrics_test.go:1089: have stack [runtime.unlock runtime.semacquire1 runtime.semacquire runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run]
        metrics_test.go:1089: have stack [runtime.unlock runtime_test.TestRuntimeLockMetricsAndProfile.func5.1 runtime_test.(*contentionWorker).run]
        metrics_test.go:1091: want stack [runtime.unlock runtime.semrelease1 runtime_test.TestRuntimeLockMetricsAndProfile.func6.1 runtime_test.(*contentionWorker).run]

— watchflakes

[gopherbot]

Change https://go.dev/cl/544375 mentions this issue: runtime: test for contention in both semaphore paths

[rhysh]

Most of the failures are from before https://go.dev/cl/544195. Following that, I've seen four failures (via fetchlogs/greplogs).

https://build.golang.org/log/c733e91b4693774eb42501bbe93a8ee071ef312e addressed by https://go.dev/cl/544375
https://build.golang.org/log/79b92d5304bd79fbdfac90c953d53b172f2d8250 addressed by https://go.dev/cl/544375
https://build.golang.org/log/2e5e4f5629a179365a02d2bb9548ed5b94597c54 during a slow runtime test which I expect adds a lot of non-contention work, making it harder to encounter the desired contention ... logging to confirm in https://go.dev/cl/544375
https://build.golang.org/log/6ee5577c3e0a08531b8d7f009631323b4aab7c23 on linux-ppc64le, showing pprof with the amount of contention I expect but a metrics view with much more weight. One of the earlier linux-390x failures shows a similar behavior (though also failed due to the big-endian bug I fixed in CL 544195).