Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

all: frequent test timeouts on Android builders #42212

Open
bcmills opened this issue Oct 26, 2020 · 8 comments
Open

all: frequent test timeouts on Android builders #42212

bcmills opened this issue Oct 26, 2020 · 8 comments
Labels
Builders x/build issues (builders, bots, dashboards) mobile Android, iOS, and x/mobile NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@bcmills bcmills added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. mobile Android, iOS, and x/mobile labels Oct 26, 2020
@gopherbot gopherbot added this to the Unreleased milestone Oct 26, 2020
@bcmills
Copy link
Member Author

bcmills commented Apr 27, 2022

greplogs --dashboard -md -l -e '\*\*\* Test killed with quit: ran too long .*\nFAIL\s+golang\.org/x/sys/unix' --since=2022-02-05

2022-04-17T00:58:13-33da011-e948c33/android-386-emu
2022-02-22T15:23:59-3681064-d17b65f/android-amd64-emu
2022-02-07T23:40:03-5739886-911c78f/android-amd64-emu

@bcmills
Copy link
Member Author

bcmills commented Apr 27, 2022

(CC @changkun, @steeve)

@changkun
Copy link
Member

changkun commented Apr 28, 2022

It looks like that tests are blocked forever at a syscall (sleep, park, etc.)

Looking at the timeline, it feels like an issue with the emulator itself. Hence I reported to Corellium, but it would not be much helpful for them to tackle down the problem, I think.

Is there anything we could really do? Perhaps mark this as unfortunate?

@bcmills
Copy link
Member Author

bcmills commented May 2, 2022

Is there anything we could really do? Perhaps mark this as unfortunate?

I can stop triaging failures on this builder, but at that point the port becomes basically abandoned unless somebody else starts triaging it. 🤷‍♂️

We get a high enough rate of these kinds of failures that they should be reproducible with enough iterations, or at least could be caught with sufficient logging and a watchdog timeout. (Maybe somebody needs to sit down with the emulator and a remote debugger and figure out what's wedged?)

@bcmills bcmills changed the title x/sys/unix: frequent test timeouts on Android builders all: frequent test timeouts on Android builders May 3, 2022
@bcmills
Copy link
Member Author

bcmills commented Jun 14, 2022

greplogs -l -e '(?ms)\Aandroid-.*\*\*\* Test killed with quit: ran too long' --since=2022-05-04
2022-06-04T16:11:54-f62514f-fc66cae/android-amd64-emu
2022-05-19T21:33:15-84e58bf-d8762b2/android-amd64-emu
2022-05-10T15:33:47-b2400d8-f7209f9/android-386-emu

@gopherbot
Copy link

gopherbot commented Jun 14, 2022

Change https://go.dev/cl/412174 mentions this issue: dashboard: add known issues for android-*-emu

gopherbot pushed a commit to golang/build that referenced this issue Jun 14, 2022
Issue golang/go#42212 manifests as test timeouts, and is by far the most
frequent of these known issues.

Issue golang/go#51001 causes failures with "systemstack called from unexpected
goroutine". It seems to have been introduced sometime last year, but
it isn't clear to me whether it is a regression or an older (latent)
bug unearthed by some other change.

Issue golang/go#52724 appears to be a bug or race in the Android emulator
itself. It might require a builder image update and/or escalation to
the maintainers of the emulator proper.

Updates golang/go#53377.

Change-Id: I677915b1ff02dd02e0f14c63b0d25caf11e27a72
Reviewed-on: https://go-review.googlesource.com/c/build/+/412174
Reviewed-by: David Chase <drchase@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) mobile Android, iOS, and x/mobile NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

3 participants