time: test timeout in TestConcurrentTimerReset{,Stop} on aix-ppc64 builder

[bcmills]

2020-03-17T00:45:15-2f54081/aix-ppc64

SIGQUIT: quit
PC=0x9000000005a2d7c m=7 sigcode=0

goroutine 0 [idle]:
runtime.osyield()
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/runtime/os2_aix.go:619 +0x44 fp=0xa0001000005e708 sp=0xa0001000005e6c8 pc=0x1000355c4
runtime.deltimer(0xa000100000aa148, 0x100269d00)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/runtime/time.go:340 +0x254 fp=0xa0001000005e730 sp=0xa0001000005e708 pc=0x10005be34
time.stopTimer(0xa000100000aa148, 0x100000000000000)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/runtime/time.go:214 +0x2c fp=0xa0001000005e760 sp=0xa0001000005e730 pc=0x10005b71c
time.(*Timer).Stop(...)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/time/sleep.go:79
time_test.TestConcurrentTimerResetStop.func2(0xa00010000400080, 0xa000100000aa140, 0x2)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/time/time_test.go:1439 +0x74 fp=0xa0001000005e7a8 sp=0xa0001000005e760 pc=0x1001b47c4
runtime.goexit()
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/runtime/asm_ppc64x.s:884 +0x4 fp=0xa0001000005e7a8 sp=0xa0001000005e7a8 pc=0x10006dbc4
created by time_test.TestConcurrentTimerResetStop
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/time/time_test.go:1437 +0xc8

goroutine 197 [running]:
runtime.asmcgocall(0x110001280, 0xa0001000005e690)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/runtime/asm_ppc64x.s:594 +0x38 fp=0xa0001000005e650 sp=0xa0001000005e630 pc=0x10006da78
runtime.syscall0(0x11002f308, 0x0, 0x0)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/runtime/os2_aix.go:187 +0x8c fp=0xa0001000005e6c8 sp=0xa0001000005e650 pc=0x1000348dc
runtime.osyield()
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/runtime/os2_aix.go:619 +0x44 fp=0xa0001000005e708 sp=0xa0001000005e6c8 pc=0x1000355c4
runtime.deltimer(0xa000100000aa148, 0x100269d00)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/runtime/time.go:340 +0x254 fp=0xa0001000005e730 sp=0xa0001000005e708 pc=0x10005be34
time.stopTimer(0xa000100000aa148, 0x100000000000000)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/runtime/time.go:214 +0x2c fp=0xa0001000005e760 sp=0xa0001000005e730 pc=0x10005b71c
time.(*Timer).Stop(...)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/time/sleep.go:79
time_test.TestConcurrentTimerResetStop.func2(0xa00010000400080, 0xa000100000aa140, 0x2)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/time/time_test.go:1439 +0x74 fp=0xa0001000005e7a8 sp=0xa0001000005e760 pc=0x1001b47c4
runtime.goexit()
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/runtime/asm_ppc64x.s:884 +0x4 fp=0xa0001000005e7a8 sp=0xa0001000005e7a8 pc=0x10006dbc4
created by time_test.TestConcurrentTimerResetStop
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/time/time_test.go:1437 +0xc8

CC @ianlancetaylor @cherrymui

[bcmills]

It's not clear to me whether this indicates a bug in the timer code, a bug in the test, or a bug in the AIX kernel, and whether it applies only to AIX or just happened to be caught there first.

The test is new in Go 1.15 (CL 221077), so marking as release-blocker until we better understand the cause.

[bcmills]

CC @trex58 @Helflym @mknyszek

[Helflym]

First, TestConcurrentTimerReset is also failing.

SIGQUIT: quit
PC=0x900000000532d3c m=0 sigcode=8

goroutine 0 [idle]:
runtime.osyield()
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/os2_aix.go:619 +0x44 fp=0xa000100003bee78 sp=0xa000100003bee38 pc=0x1000355c4
runtime.modtimer(0xa00010000164058, 0x15fd33f628503228, 0x0, 0x100230170, 0x1001d4100, 0xa0001000010fb00, 0x0)
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/time.go:433 +0x35c fp=0xa000100003beeb0 sp=0xa000100003bee78 pc=0x10005c59c
runtime.resettimer(...)
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/time.go:494
time.resetTimer(0xa00010000164058, 0x15fd33f628503228)
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/time.go:223 +0x68 fp=0xa000100003bef08 sp=0xa000100003beeb0 pc=0x10005b7a8
time.(*Timer).Reset(0xa00010000164050, 0x34630b8a948, 0x100000000000000)
        /opt/freeware/src/packages/BUILD/go-root/src/time/sleep.go:127 +0x80 fp=0xa000100003bef50 sp=0xa000100003bef08 pc=0x100089820
time_test.TestConcurrentTimerReset.func1(0xa00010000114060, 0xa00010000164050, 0x4)
        /opt/freeware/src/packages/BUILD/go-root/src/time/time_test.go:1416 +0x7c fp=0xa000100003befa8 sp=0xa000100003bef50 pc=0x1001b464c
runtime.goexit()
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/asm_ppc64x.s:884 +0x4 fp=0xa000100003befa8 sp=0xa000100003befa8 pc=0x10006dbc4
created by time_test.TestConcurrentTimerReset
        /opt/freeware/src/packages/BUILD/go-root/src/time/time_test.go:1413 +0x98

goroutine 180 [running]:
runtime.asmcgocall(0x110000d80, 0xa000100003bee00)
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/asm_ppc64x.s:594 +0x38 fp=0xa000100003bedc0 sp=0xa000100003beda0 pc=0x10006da78
runtime.syscall0(0x11002ee08, 0x0, 0x0)
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/os2_aix.go:187 +0x8c fp=0xa000100003bee38 sp=0xa000100003bedc0 pc=0x1000348dc
runtime.osyield()
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/os2_aix.go:619 +0x44 fp=0xa000100003bee78 sp=0xa000100003bee38 pc=0x1000355c4
runtime.modtimer(0xa00010000164058, 0x15fd33f628503228, 0x0, 0x100230170, 0x1001d4100, 0xa0001000010fb00, 0x0)
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/time.go:433 +0x35c fp=0xa000100003beeb0 sp=0xa000100003bee78 pc=0x10005c59c
runtime.resettimer(...)
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/time.go:494
time.resetTimer(0xa00010000164058, 0x15fd33f628503228)
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/time.go:223 +0x68 fp=0xa000100003bef08 sp=0xa000100003beeb0 pc=0x10005b7a8
time.(*Timer).Reset(0xa00010000164050, 0x34630b8a948, 0x100000000000000)
        /opt/freeware/src/packages/BUILD/go-root/src/time/sleep.go:127 +0x80 fp=0xa000100003bef50 sp=0xa000100003bef08 pc=0x100089820
time_test.TestConcurrentTimerReset.func1(0xa00010000114060, 0xa00010000164050, 0x4)
        /opt/freeware/src/packages/BUILD/go-root/src/time/time_test.go:1416 +0x7c fp=0xa000100003befa8 sp=0xa000100003bef50 pc=0x1001b464c
runtime.goexit()
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/asm_ppc64x.s:884 +0x4 fp=0xa000100003befa8 sp=0xa000100003befa8 pc=0x10006dbc4
created by time_test.TestConcurrentTimerReset
        /opt/freeware/src/packages/BUILD/go-root/src/time/time_test.go:1413 +0x98

goroutine 181 [runnable]:
runtime.resettimer(...)
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/time.go:494
time.resetTimer(0xa00010000164058, 0x15fd33f62852e800)
        /opt/freeware/src/packages/BUILD/go-root/src/runtime/time.go:223 +0x68
time.(*Timer).Reset(0xa00010000164050, 0x34630b8a000, 0x0)
        /opt/freeware/src/packages/BUILD/go-root/src/time/sleep.go:127 +0x80
time_test.TestConcurrentTimerReset.func1(0xa00010000114060, 0xa00010000164050, 0x5)
        /opt/freeware/src/packages/BUILD/go-root/src/time/time_test.go:1416 +0x7c
created by time_test.TestConcurrentTimerReset
        /opt/freeware/src/packages/BUILD/go-root/src/time/time_test.go:1413 +0x98

Secondly, I've managed to reproduce it when launching the whole time testsuite, but not when I'm launching only one of TestConcurrentTimerReset tests.

Finally, it has failed on AIX 7.2TL0 and 7.2TL2, but not yet on 7.2TL3, so it might indeed be a kernel bug.

[bcmills]

A TestConccurrentTimerReset in the builder:
2020-03-23T21:15:27-fb2a634/aix-ppc64

[gopherbot]

Change https://golang.org/cl/225497 mentions this issue: runtime: prevent preemption while timer is in timerModifying

[gopherbot]

Change https://golang.org/cl/225521 mentions this issue: runtime: prevent preemption while timer is in timerModifying

[dmitshur]

The status on March 16th was:

It's not clear to me whether this indicates a bug in the timer code, a bug in the test, or a bug in the AIX kernel, and whether it applies only to AIX or just happened to be caught there first.

There've been some additional test failure instances, findings, and @mknyszek's CL 225497 mentioning this issue since then.

Has that helped narrow down where the problem may be, or does this issue still need more investigation?

[ianlancetaylor]

This hasn't happened since March 23, and it was plausibly fixed by the CLs linked above, so I'm calling this fixed.