Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/benchmarks/sweet/cmd/sweet: timeout in TestSweetEndToEnd #51052

Open
bcmills opened this issue Feb 7, 2022 · 5 comments
Open

x/benchmarks/sweet/cmd/sweet: timeout in TestSweetEndToEnd #51052

bcmills opened this issue Feb 7, 2022 · 5 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@bcmills
Copy link
Member

bcmills commented Feb 7, 2022

greplogs --dashboard -md -l -e 'panic: test timed out .*\n(?:\ngoroutine \d+ .*:\n(.+\n)*)*^golang\.org/x/benchmarks/sweet/cmd/sweet_test.TestSweetEndToEnd' --since=2022-01-01

2022-02-07T07:14:28-e4ea61d-334a591/linux-amd64-longtest


(attn @mknyszek)

@gopherbot gopherbot added this to the Unreleased milestone Feb 7, 2022
@mknyszek
Copy link
Contributor

mknyszek commented Feb 7, 2022

That's very odd. Other successful runs of this test take around 3-4 minutes. Based on the output, this looks like maybe a deadlock in bleve?

@bcmills
Copy link
Member Author

bcmills commented Feb 7, 2022

There are some bleve goroutines left running, but OTOH TestSweetEndToEnd seems to run all of the actual tests as subprocess, and there is definitely at least one subprocess still running at the time of the timeout too. I think those goroutines might be a red herring, although they may point to a dependency init issue elsewhere in the package.

I suspect that the timeout is due to a subprocess (compare #50436):

goroutine 20 [syscall, 10 minutes]:
syscall.Syscall6(0xf7, 0x1, 0x1978, 0xc0000dd9c0, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/asm_linux_amd64.s:43 +0x5
os.(*Process).blockUntilWaitable(0xc0002dc000)
	/workdir/go/src/os/wait_waitid.go:32 +0x9c
os.(*Process).wait(0xc0002dc000)
	/workdir/go/src/os/exec_unix.go:22 +0x28
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:132
os/exec.(*Cmd).Wait(0xc0001c8000)
	/workdir/go/src/os/exec/exec.go:507 +0x54
os/exec.(*Cmd).Run(0x8?)
	/workdir/go/src/os/exec/exec.go:341 +0x39
os/exec.(*Cmd).CombinedOutput(0xc0001c8000)
	/workdir/go/src/os/exec/exec.go:567 +0x96
golang.org/x/benchmarks/sweet/cmd/sweet_test.TestSweetEndToEnd(0xc000102b60)
	/workdir/gopath/src/golang.org/x/benchmarks/sweet/cmd/sweet/integration_test.go:89 +0x930
testing.tRunner(0xc000102b60, 0xc4b290)
	/workdir/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
	/workdir/go/src/testing/testing.go:1486 +0x35f

@bcmills
Copy link
Member Author

bcmills commented Feb 7, 2022

@mknyszek
Copy link
Contributor

mknyszek commented Feb 7, 2022

I think those goroutines might be a red herring, although they may point to a dependency init issue elsewhere in the package.

Yeah, that makes sense. I mistakenly thought I'd landed the change that dumps subprocess output. I did not.

Since it's sweet get, that most likely points to a network flake of some kind. Any suggestions on how I can make this test more resilient to those?

@bcmills
Copy link
Member Author

bcmills commented Feb 7, 2022

Since it's sweet get, that most likely points to a network flake of some kind. Any suggestions on how I can make this test more resilient to those?

Mainly, plumb through Contexts everywhere, so that if things are broken you can have the subprocess dump useful errors before the text is terminated.

You can use (*testing.T).Deadline to get the test's deadline, and then subtract off some amount of headroom (for starting, communicating with, and cleaning up the subprocess). Then you can plumb in the resulting duration as a flag.Duration for a -timeout flag. (#50436 plus signal.NotifyContext would get you most of the way to using signals instead of flags, but signals aren't really viable on Windows anyway. 🤷‍♂️)

@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants