Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: frequent test timeouts on solaris-amd64-oraclerel #51443

Open
bcmills opened this issue Mar 2, 2022 · 11 comments
Open

runtime: frequent test timeouts on solaris-amd64-oraclerel #51443

bcmills opened this issue Mar 2, 2022 · 11 comments
Labels
Builders x/build issues (builders, bots, dashboards) compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Solaris
Milestone

Comments

@bcmills
Copy link
Member

bcmills commented Mar 2, 2022

greplogs --dashboard -md -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-01-01

2022-03-01T21:27:42-b0db2f0/solaris-amd64-oraclerel
2022-02-28T19:00:23-b33592d/solaris-amd64-oraclerel
2022-01-08T00:24:25-90860e0/solaris-amd64-oraclerel
2022-01-06T23:39:43-042548b/solaris-amd64-oraclerel

@golang/runtime: the builder appears to only run the -short tests. Is there something we can feasibly do to make -short mode shorter?

@golang/release, @rorth: would it make sense to set GO_TEST_TIMEOUT_SCALE on this builder to make the timeouts more generous?

@bcmills bcmills added Builders x/build issues (builders, bots, dashboards) OS-Solaris NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 2, 2022
@bcmills bcmills added this to the Backlog milestone Mar 2, 2022
@dmitshur
Copy link
Contributor

dmitshur commented Mar 2, 2022

It makes sense to increase GO_TEST_TIMEOUT_SCALE to me, but @rorth as the builder owner is in a better position to decide the optimal configuration for this builder. This builder is not configured as a TryBot, so if it needs more time, it's fine and won't affect pre-submit testing times (#17104).

@rorth
Copy link

rorth commented Mar 3, 2022

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Mar 4, 2022

While running tests, a go builder will happily grab whatever resources it has available. Are you doing anything to restrict it to 4 jobs?

@rorth
Copy link

rorth commented Mar 4, 2022

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Mar 4, 2022

Good point, I think the builder config will set GOMAXPROCS, which will limit the resource usage of each specific test, but then you will have 4 tests run in parallel and each test will happily run 4 tests in parallel. At least, I think that is how it works; my apologies if I'm getting this wrong.

@rorth
Copy link

rorth commented Mar 7, 2022

@bcmills
Copy link
Member Author

bcmills commented Mar 30, 2022

As a first step, I've now set GOMAXPROCS=8 for the builder to see if
this helps. The last runtime timeout happened before that change,
though. We'll have to see if this is enough.

Appears not to be:

greplogs --dashboard -md -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-03-08

2022-03-29T16:24:51-a2baae6/solaris-amd64-oraclerel
2022-03-25T19:04:59-2bbf383/solaris-amd64-oraclerel
2022-03-25T15:15:57-3fd8b86/solaris-amd64-oraclerel

@bcmills
Copy link
Member Author

bcmills commented May 3, 2022

greplogs --dashboard -md -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-03-30
2022-05-02T19:53:56-3b01a80/solaris-amd64-oraclerel

@bcmills
Copy link
Member Author

bcmills commented May 11, 2022

greplogs -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-05-03
2022-05-11T03:28:01-ccb7987/solaris-amd64-oraclerel
2022-05-10T21:56:21-bda9da8/solaris-amd64-oraclerel

@bcmills
Copy link
Member Author

bcmills commented May 26, 2022

greplogs -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-05-12
2022-05-25T19:25:08-04337a6/solaris-amd64-oraclerel
2022-05-23T16:21:22-c1d197a/solaris-amd64-oraclerel

@gopherbot
Copy link

gopherbot commented May 26, 2022

Change https://go.dev/cl/408701 mentions this issue: dashboard: add known issue for solaris-amd64-oraclerel

gopherbot pushed a commit to golang/build that referenced this issue May 26, 2022
For golang/go#52653.
Updates golang/go#51443.

Change-Id: Iaea8fab13ed979e54c827f0f3c4d705bdaff4ee4
Reviewed-on: https://go-review.googlesource.com/c/build/+/408701
Reviewed-by: Alex Rakoczy <alex@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Solaris
Projects
Status: Triage Backlog
Development

No branches or pull requests

5 participants