runtime: timeouts in buildTestProg on linux-ppc64le-buildlet

[bcmills]

2021-02-19T00:04:42-1c659f2/linux-ppc64le-buildlet
2021-02-19T00:04:30-b6379f1/linux-ppc64le-buildlet
2021-02-19T00:04:22-09e059a/linux-ppc64le-buildlet
2021-02-19T00:02:06-01f05d8/linux-ppc64le-buildlet
2021-02-19T00:01:17-e7ee3c1/linux-ppc64le-buildlet
2021-02-16T22:18:37-8482559/linux-ppc64le-buildlet
2020-12-15T21:45:05-731bb54/linux-ppc64le-buildlet
2020-12-15T21:04:49-129bb19/linux-ppc64le-buildlet
2020-12-15T21:01:37-685a322/linux-ppc64le-buildlet
2020-12-15T20:58:17-3d64678/linux-ppc64le-buildlet
2020-12-15T20:55:01-7cdc84a/linux-ppc64le-buildlet
2020-12-14T22:39:04-663cd86/linux-ppc64le-buildlet
2020-12-14T21:09:33-d06794d/linux-ppc64le-buildlet
2020-12-14T18:06:06-828746e/linux-ppc64le-buildlet

It's not clear to me whether the test is hung or just slow. Perhaps we should set a GO_TEST_TIMEOUT_SCALE on this builder and see if that helps?

CC @golang/release @bradfitz @laboger @ceseo

[laboger]

I've been looking at this problem on various systems. If I run 'go tool dist test go_test:runtime' by itself the time varies and sometimes on a power8 it times out, but never on a power9 because it is faster there. But one mysterious result is that if I set GOMAXPROCS to a smaller value, like 2, 4, or 8 then the time is small enough on the power8 so that it doesn't time out. I thought these VMs only had 2 processors each, which I thought was equivalent to running GOMAXPROCS=2. I can only get it to timeout if the number of processors is > 150.

[ceseo]

I don't know if this is related, but a while back I had a bunch of timeouts in my personal OSU power9 system. I asked them to increase it from 2 to 4 cores and 4 to 8GB RAM and now I have zero timeouts there. Maybe these builders are timing out because they have too little CPU time allocated to them in the hypervisor.

@bradfitz can correct me if I'm wrong, but IIRC, the power8 builders are 2 core, 4GB RAM only.

[laboger]

From golang.org/x/build/env/linux-ppc64le/osuosl/NOTES: * go-le-bionic-1: (20 GB RAM, 50 GB disk, 10 cores, POWER9) Linux go-le-bionic-1 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux * go-le-bionic-2: (20 GB RAM, 50 GB disk, 10 cores, POWER8) Linux go-le-bionic-2 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

…

On Tue, Feb 23, 2021 at 2:08 PM Carlos Eduardo Seo ***@***.***> wrote: I don't know if this is related, but a while back I had a bunch of timeouts in my personal OSU power9 system. I asked them to increase it from 2 to 4 cores and 4 to 8GB RAM and now I have zero timeouts there. Maybe these builders are timing out because they have too little CPU time allocated to them in the hypervisor. @bradfitz <https://github.com/bradfitz> can correct me if I'm wrong, but IIRC, the power8 builders are 2 core, 4GB RAM only. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44422 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACH7BDGWKU6CUMSDJWWFK3TTAQDM5ANCNFSM4X4PGX5A> .