runtime: timeouts in buildTestProg on linux-ppc64le-buildlet #44422
Comments
I've been looking at this problem on various systems. If I run 'go tool dist test go_test:runtime' by itself the time varies and sometimes on a power8 it times out, but never on a power9 because it is faster there. But one mysterious result is that if I set GOMAXPROCS to a smaller value, like 2, 4, or 8 then the time is small enough on the power8 so that it doesn't time out. I thought these VMs only had 2 processors each, which I thought was equivalent to running GOMAXPROCS=2. I can only get it to timeout if the number of processors is > 150. |
I don't know if this is related, but a while back I had a bunch of timeouts in my personal OSU power9 system. I asked them to increase it from 2 to 4 cores and 4 to 8GB RAM and now I have zero timeouts there. Maybe these builders are timing out because they have too little CPU time allocated to them in the hypervisor. @bradfitz can correct me if I'm wrong, but IIRC, the power8 builders are 2 core, 4GB RAM only. |
From golang.org/x/build/env/linux-ppc64le/osuosl/NOTES:
* go-le-bionic-1: (20 GB RAM, 50 GB disk, 10 cores, POWER9)
Linux go-le-bionic-1 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54
UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
* go-le-bionic-2: (20 GB RAM, 50 GB disk, 10 cores, POWER8)
Linux go-le-bionic-2 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54
UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
…On Tue, Feb 23, 2021 at 2:08 PM Carlos Eduardo Seo ***@***.***> wrote:
I don't know if this is related, but a while back I had a bunch of
timeouts in my personal OSU power9 system. I asked them to increase it from
2 to 4 cores and 4 to 8GB RAM and now I have zero timeouts there. Maybe
these builders are timing out because they have too little CPU time
allocated to them in the hypervisor.
@bradfitz <https://github.com/bradfitz> can correct me if I'm wrong, but
IIRC, the power8 builders are 2 core, 4GB RAM only.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#44422 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACH7BDGWKU6CUMSDJWWFK3TTAQDM5ANCNFSM4X4PGX5A>
.
|
2021-02-19T00:04:42-1c659f2/linux-ppc64le-buildlet
2021-02-19T00:04:30-b6379f1/linux-ppc64le-buildlet
2021-02-19T00:04:22-09e059a/linux-ppc64le-buildlet
2021-02-19T00:02:06-01f05d8/linux-ppc64le-buildlet
2021-02-19T00:01:17-e7ee3c1/linux-ppc64le-buildlet
2021-02-16T22:18:37-8482559/linux-ppc64le-buildlet
2020-12-15T21:45:05-731bb54/linux-ppc64le-buildlet
2020-12-15T21:04:49-129bb19/linux-ppc64le-buildlet
2020-12-15T21:01:37-685a322/linux-ppc64le-buildlet
2020-12-15T20:58:17-3d64678/linux-ppc64le-buildlet
2020-12-15T20:55:01-7cdc84a/linux-ppc64le-buildlet
2020-12-14T22:39:04-663cd86/linux-ppc64le-buildlet
2020-12-14T21:09:33-d06794d/linux-ppc64le-buildlet
2020-12-14T18:06:06-828746e/linux-ppc64le-buildlet
It's not clear to me whether the test is hung or just slow. Perhaps we should set a
GO_TEST_TIMEOUT_SCALE
on this builder and see if that helps?CC @golang/release @bradfitz @laboger @ceseo
The text was updated successfully, but these errors were encountered: