runtime: frequent test timeouts on solaris-amd64-oraclerel #51443
Comments
bcmills
added
Builders
|
It makes sense to increase |
|
"Bryan C. Mills" ***@***.***> writes:
@golang/release, @rorth: would it make sense to set `GO_TEST_TIMEOUT_SCALE`
on this builder to make the timeouts more generous?
I'd be ok with that, but wondered about one thing: that zone is shared
between the golang builder and an LLVM buildbot. The latter is
restricted to 8 parallel jobs, and I'd been under the impression that
the golang one would work with just 4 jobs. However, recently I've
often seen way more than those 4 jobs. Have I just been deluded about
that restriction, or has it somehow been lost?
|
|
While running tests, a go builder will happily grab whatever resources it has available. Are you doing anything to restrict it to 4 jobs? |
|
Hi Ian,
While running tests, a go builder will happily grab whatever resources it
has available. Are you doing anything to restrict it to 4 jobs?
Not so far. When I inherited the builder, I was under the (obviously
wrong) impression that this had been done in the builder config,
similarly to buildbot for LLVM. So there's no similar facility
available?
I should be able to use CPU shares to achieve this. Setting up a
separate zone just for the golang builder seems like a waste of effort
(and IPv4 addresses).
|
|
Good point, I think the builder config will set |
|
Ian Lance Taylor ***@***.***> writes:
Good point, I think the builder config will set `GOMAXPROCS`, which will
limit the resource usage of each specific test, but then you will have 4
tests run in parallel and each test will happily run 4 tests in parallel.
At least, I think that is how it works; my apologies if I'm getting this
wrong.
As a first step, I've now set GOMAXPROCS=8 for the builder to see if
this helps. The last runtime timeout happened before that change,
though. We'll have to see if this is enough.
In parallel, I'm also running the runtime test 10000 times in a loop on
identical hardware (although on bare metal instead of a kernel
zone/VM). Of the roughly 4000 runs completed so far, 6 have taken 2+
minutes rather than the usual 16-18 seconds. I missed the
-test.timeout=3m, unfortunately, so all tests PASSed.
Unfortunately, I couldn't easily see which subtest is timing out,
otherwise I could restrict the testing to just that one if it's always
the same one.
|
Appears not to be:
2022-03-29T16:24:51-a2baae6/solaris-amd64-oraclerel |
|
|
|
|
greplogs --dashboard -md -l -e '(?ms)\Asolaris-amd64-oraclerel.*^panic: test timed out.*FAIL\s+runtime' --since=2022-01-012022-03-01T21:27:42-b0db2f0/solaris-amd64-oraclerel
2022-02-28T19:00:23-b33592d/solaris-amd64-oraclerel
2022-01-08T00:24:25-90860e0/solaris-amd64-oraclerel
2022-01-06T23:39:43-042548b/solaris-amd64-oraclerel
@golang/runtime: the builder appears to only run the
-shorttests. Is there something we can feasibly do to make-shortmode shorter?@golang/release, @rorth: would it make sense to set
GO_TEST_TIMEOUT_SCALEon this builder to make the timeouts more generous?The text was updated successfully, but these errors were encountered: