Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: frequent test timeouts on solaris-amd64-oraclerel #51443

Open
bcmills opened this issue Mar 2, 2022 · 9 comments
Open

runtime: frequent test timeouts on solaris-amd64-oraclerel #51443

bcmills opened this issue Mar 2, 2022 · 9 comments
Labels
Builders NeedsInvestigation OS-Solaris
Milestone

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 2, 2022

greplogs --dashboard -md -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-01-01

2022-03-01T21:27:42-b0db2f0/solaris-amd64-oraclerel
2022-02-28T19:00:23-b33592d/solaris-amd64-oraclerel
2022-01-08T00:24:25-90860e0/solaris-amd64-oraclerel
2022-01-06T23:39:43-042548b/solaris-amd64-oraclerel

@golang/runtime: the builder appears to only run the -short tests. Is there something we can feasibly do to make -short mode shorter?

@golang/release, @rorth: would it make sense to set GO_TEST_TIMEOUT_SCALE on this builder to make the timeouts more generous?

@bcmills bcmills added Builders OS-Solaris NeedsInvestigation labels Mar 2, 2022
@bcmills bcmills added this to the Backlog milestone Mar 2, 2022
@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Mar 2, 2022

It makes sense to increase GO_TEST_TIMEOUT_SCALE to me, but @rorth as the builder owner is in a better position to decide the optimal configuration for this builder. This builder is not configured as a TryBot, so if it needs more time, it's fine and won't affect pre-submit testing times (#17104).

@rorth
Copy link

@rorth rorth commented Mar 3, 2022

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Mar 4, 2022

While running tests, a go builder will happily grab whatever resources it has available. Are you doing anything to restrict it to 4 jobs?

@rorth
Copy link

@rorth rorth commented Mar 4, 2022

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Mar 4, 2022

Good point, I think the builder config will set GOMAXPROCS, which will limit the resource usage of each specific test, but then you will have 4 tests run in parallel and each test will happily run 4 tests in parallel. At least, I think that is how it works; my apologies if I'm getting this wrong.

@rorth
Copy link

@rorth rorth commented Mar 7, 2022

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 30, 2022

As a first step, I've now set GOMAXPROCS=8 for the builder to see if
this helps. The last runtime timeout happened before that change,
though. We'll have to see if this is enough.

Appears not to be:

greplogs --dashboard -md -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-03-08

2022-03-29T16:24:51-a2baae6/solaris-amd64-oraclerel
2022-03-25T19:04:59-2bbf383/solaris-amd64-oraclerel
2022-03-25T15:15:57-3fd8b86/solaris-amd64-oraclerel

@bcmills
Copy link
Member Author

@bcmills bcmills commented May 3, 2022

greplogs --dashboard -md -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-03-30
2022-05-02T19:53:56-3b01a80/solaris-amd64-oraclerel

@bcmills
Copy link
Member Author

@bcmills bcmills commented May 11, 2022

greplogs -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-05-03
2022-05-11T03:28:01-ccb7987/solaris-amd64-oraclerel
2022-05-10T21:56:21-bda9da8/solaris-amd64-oraclerel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders NeedsInvestigation OS-Solaris
Projects
None yet
Development

No branches or pull requests

4 participants