Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: test timeouts on linux-ppc64le-buildlet #62272

Closed
gopherbot opened this issue Aug 24, 2023 · 6 comments
Closed

runtime: test timeouts on linux-ppc64le-buildlet #62272

gopherbot opened this issue Aug 24, 2023 · 6 comments
Labels
arch-ppc64x Builders x/build issues (builders, bots, dashboards) compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@gopherbot
Copy link
Contributor

gopherbot commented Aug 24, 2023

#!watchflakes
post <- goarch == "ppc64le" && pkg == "runtime" && (`test timed out` || `panic: finalizer started .* ago .* and ran \d+ iterations, but did not return` || `failed to start: context deadline exceeded` || `WaitDelay expired before I/O complete`) && date < "2023-10-01"

Issue created automatically to collect these failures.

Example (log):

panic: test timed out after 6m0s
running tests:
	TestCgoPprofPIE (21s)
	TestDeferKeepAlive (1s)
	TestGcPacer (1s)
	TestGdbAutotmpTypes (5s)
	TestGdbPythonCgo (3s)

runtime.gopark(0x418f7a4d197a5a?, 0xc000137eb8?, 0xc8?, 0x7e?, 0x55a?)
	/workdir/go/src/runtime/proc.go:398 +0x114 fp=0xc000137e38 sp=0xc000137e08 pc=0x56094
time.Sleep(0x3b9aca00)
	/workdir/go/src/runtime/time.go:195 +0x120 fp=0xc000137e98 sp=0xc000137e38 pc=0x91cd0
runtime_test.TestDeferKeepAlive(0xc0003ec820)
	/workdir/go/src/runtime/mfinal_test.go:246 +0x150 fp=0xc000137f30 sp=0xc000137e98 pc=0x341330
testing.tRunner(0xc0003ec820, 0x4962a0)

watchflakes

@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 24, 2023
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "runtime" && test == "TestDeferKeepAlive"
2023-08-24 18:14 linux-ppc64le-buildlet go@f0b05092 runtime.TestDeferKeepAlive (log)
panic: test timed out after 6m0s
running tests:
	TestCgoPprofPIE (21s)
	TestDeferKeepAlive (1s)
	TestGcPacer (1s)
	TestGdbAutotmpTypes (5s)
	TestGdbPythonCgo (3s)

runtime.gopark(0x418f7a4d197a5a?, 0xc000137eb8?, 0xc8?, 0x7e?, 0x55a?)
	/workdir/go/src/runtime/proc.go:398 +0x114 fp=0xc000137e38 sp=0xc000137e08 pc=0x56094
time.Sleep(0x3b9aca00)
	/workdir/go/src/runtime/time.go:195 +0x120 fp=0xc000137e98 sp=0xc000137e38 pc=0x91cd0
runtime_test.TestDeferKeepAlive(0xc0003ec820)
	/workdir/go/src/runtime/mfinal_test.go:246 +0x150 fp=0xc000137f30 sp=0xc000137e98 pc=0x341330
testing.tRunner(0xc0003ec820, 0x4962a0)

watchflakes

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Aug 24, 2023
@bcmills bcmills changed the title runtime: TestDeferKeepAlive failures x/build: test timeouts on linux-ppc64le-buildlet Aug 25, 2023
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Aug 25, 2023
@gopherbot gopherbot added this to the Unreleased milestone Aug 25, 2023
@bcmills bcmills changed the title x/build: test timeouts on linux-ppc64le-buildlet runtime: test timeouts on linux-ppc64le-buildlet Aug 25, 2023
@bcmills bcmills modified the milestones: Unreleased, Backlog Aug 25, 2023
@bcmills
Copy link
Contributor

bcmills commented Aug 25, 2023

The tests that were running don't seem to be stuck anywhere.

It's not obvious to me whether the runtime tests have gotten too slow in -short mode, or the builder just needs a longer GO_TEST_TIMEOUT_SCALE (see previously #44422) or lower test parallelism (e.g. to avoid hitting swap).

@golang/ppc64, could a builder owner investigate?

(CC @golang/runtime)

@pmur
Copy link
Contributor

pmur commented Aug 25, 2023

I have noticed intermittent slow downs on the power8 VMs. This was the case around the failing time. I did take a cursory look shortly after the failure. I suspect this depends on the load on unrelated VMs.

If we start seeing this too often, we can try bumping the scale for the P8 builders.

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 25, 2023
@gopherbot
Copy link
Contributor Author

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 25, 2023
@gopherbot gopherbot reopened this Oct 10, 2023
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goarch == "ppc64le" && pkg == "runtime" && (`test timed out` || `panic: finalizer started .* ago .* and ran \d+ iterations, but did not return` || `failed to start: context deadline exceeded` || `WaitDelay expired before I/O complete`)
2023-10-10 00:25 linux-ppc64le-power9osu go@b201cf4a runtime.TestGdbPython (log)
panic: test timed out after 6m0s
running tests:
	TestGdbPython (5m13s)

syscall.Syscall6(0x110, 0x1, 0xc223, 0xc0001d5740, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0001d56e8 sp=0xc0001d5678 pc=0xabb78
os.(*Process).blockUntilWaitable(0xc00040ec90)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc0001d57d0 sp=0xc0001d56e8 pc=0xd592c
os.(*Process).wait(0xc00040ec90)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc0001d5850 sp=0xc0001d57d0 pc=0xcfb5c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0005f2f20)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0001d58e0 sp=0xc0001d5850 pc=0x164ab4
os/exec.(*Cmd).Run(0xc0005f2f20)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0001d5908 sp=0xc0001d58e0 pc=0x1638e0
os/exec.(*Cmd).CombinedOutput(0xc0005f2f20)
	/workdir/go/src/os/exec/exec.go:1012 +0xa0 fp=0xc0001d5960 sp=0xc0001d5908 pc=0x165390
runtime_test.testGdbPython(0xc0002849c0, 0x0)
	/workdir/go/src/runtime/runtime-gdb_test.go:272 +0xdec fp=0xc0001d5f00 sp=0xc0001d5960 pc=0x38580c
runtime_test.TestGdbPython(0x0?)
	/workdir/go/src/runtime/runtime-gdb_test.go:160 +0x30 fp=0xc0001d5f30 sp=0xc0001d5f00 pc=0x384990
testing.tRunner(0xc0002849c0, 0x4a5aa0)

watchflakes

@bcmills
Copy link
Contributor

bcmills commented Oct 12, 2023

The above looks like a separate failure mode that should be reported as a new bug. Added a date constraint to the watchflakes pattern.

@bcmills bcmills closed this as completed Oct 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-ppc64x Builders x/build issues (builders, bots, dashboards) compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Archived in project
Development

No branches or pull requests

3 participants