runtime: frequent test timeouts on solaris-amd64-oraclerel

[bcmills]

greplogs --dashboard -md -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-01-01

2022-03-01T21:27:42-b0db2f0/solaris-amd64-oraclerel
2022-02-28T19:00:23-b33592d/solaris-amd64-oraclerel
2022-01-08T00:24:25-90860e0/solaris-amd64-oraclerel
2022-01-06T23:39:43-042548b/solaris-amd64-oraclerel

@golang/runtime: the builder appears to only run the -short tests. Is there something we can feasibly do to make -short mode shorter?

@golang/release, @rorth: would it make sense to set GO_TEST_TIMEOUT_SCALE on this builder to make the timeouts more generous?

[dmitshur]

It makes sense to increase GO_TEST_TIMEOUT_SCALE to me, but @rorth as the builder owner is in a better position to decide the optimal configuration for this builder. This builder is not configured as a TryBot, so if it needs more time, it's fine and won't affect pre-submit testing times (#17104).

[rorth]

"Bryan C. Mills" ***@***.***> writes:

@golang/release, @rorth: would it make sense to set `GO_TEST_TIMEOUT_SCALE` on this builder to make the timeouts more generous?

I'd be ok with that, but wondered about one thing: that zone is shared between the golang builder and an LLVM buildbot. The latter is restricted to 8 parallel jobs, and I'd been under the impression that the golang one would work with just 4 jobs. However, recently I've often seen way more than those 4 jobs. Have I just been deluded about that restriction, or has it somehow been lost?

[ianlancetaylor]

While running tests, a go builder will happily grab whatever resources it has available. Are you doing anything to restrict it to 4 jobs?

[rorth]

Hi Ian,

While running tests, a go builder will happily grab whatever resources it has available. Are you doing anything to restrict it to 4 jobs?

Not so far. When I inherited the builder, I was under the (obviously wrong) impression that this had been done in the builder config, similarly to buildbot for LLVM. So there's no similar facility available? I should be able to use CPU shares to achieve this. Setting up a separate zone just for the golang builder seems like a waste of effort (and IPv4 addresses).

[ianlancetaylor]

Good point, I think the builder config will set GOMAXPROCS, which will limit the resource usage of each specific test, but then you will have 4 tests run in parallel and each test will happily run 4 tests in parallel. At least, I think that is how it works; my apologies if I'm getting this wrong.

[rorth]

Ian Lance Taylor ***@***.***> writes:

Good point, I think the builder config will set `GOMAXPROCS`, which will limit the resource usage of each specific test, but then you will have 4 tests run in parallel and each test will happily run 4 tests in parallel. At least, I think that is how it works; my apologies if I'm getting this wrong.

As a first step, I've now set GOMAXPROCS=8 for the builder to see if this helps. The last runtime timeout happened before that change, though. We'll have to see if this is enough. In parallel, I'm also running the runtime test 10000 times in a loop on identical hardware (although on bare metal instead of a kernel zone/VM). Of the roughly 4000 runs completed so far, 6 have taken 2+ minutes rather than the usual 16-18 seconds. I missed the -test.timeout=3m, unfortunately, so all tests PASSed. Unfortunately, I couldn't easily see which subtest is timing out, otherwise I could restrict the testing to just that one if it's always the same one.

[bcmills]

As a first step, I've now set GOMAXPROCS=8 for the builder to see if
this helps. The last runtime timeout happened before that change,
though. We'll have to see if this is enough.

Appears not to be:

greplogs --dashboard -md -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-03-08

2022-03-29T16:24:51-a2baae6/solaris-amd64-oraclerel
2022-03-25T19:04:59-2bbf383/solaris-amd64-oraclerel
2022-03-25T15:15:57-3fd8b86/solaris-amd64-oraclerel

[bcmills]

greplogs --dashboard -md -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-03-30
2022-05-02T19:53:56-3b01a80/solaris-amd64-oraclerel

[bcmills]

greplogs -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-05-03
2022-05-11T03:28:01-ccb7987/solaris-amd64-oraclerel
2022-05-10T21:56:21-bda9da8/solaris-amd64-oraclerel

[bcmills]

greplogs -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-05-12
2022-05-25T19:25:08-04337a6/solaris-amd64-oraclerel
2022-05-23T16:21:22-c1d197a/solaris-amd64-oraclerel

[gopherbot]

Change https://go.dev/cl/408701 mentions this issue: dashboard: add known issue for solaris-amd64-oraclerel