runtime: timeouts in buildTestProg #44422

bcmills · 2021-02-19T14:37:34Z

It's not clear to me whether the test is hung or just slow. Perhaps we should set a GO_TEST_TIMEOUT_SCALE on this builder and see if that helps?

CC @golang/release @bradfitz @laboger @ceseo

The text was updated successfully, but these errors were encountered:

laboger · 2021-02-23T17:48:53Z

I've been looking at this problem on various systems. If I run 'go tool dist test go_test:runtime' by itself the time varies and sometimes on a power8 it times out, but never on a power9 because it is faster there. But one mysterious result is that if I set GOMAXPROCS to a smaller value, like 2, 4, or 8 then the time is small enough on the power8 so that it doesn't time out. I thought these VMs only had 2 processors each, which I thought was equivalent to running GOMAXPROCS=2. I can only get it to timeout if the number of processors is > 150.

ceseo · 2021-02-23T20:08:30Z

I don't know if this is related, but a while back I had a bunch of timeouts in my personal OSU power9 system. I asked them to increase it from 2 to 4 cores and 4 to 8GB RAM and now I have zero timeouts there. Maybe these builders are timing out because they have too little CPU time allocated to them in the hypervisor.

@bradfitz can correct me if I'm wrong, but IIRC, the power8 builders are 2 core, 4GB RAM only.

laboger · 2021-02-24T17:05:07Z

From golang.org/x/build/env/linux-ppc64le/osuosl/NOTES: * go-le-bionic-1: (20 GB RAM, 50 GB disk, 10 cores, POWER9) Linux go-le-bionic-1 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux * go-le-bionic-2: (20 GB RAM, 50 GB disk, 10 cores, POWER8) Linux go-le-bionic-2 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

…

On Tue, Feb 23, 2021 at 2:08 PM Carlos Eduardo Seo ***@***.***> wrote: I don't know if this is related, but a while back I had a bunch of timeouts in my personal OSU power9 system. I asked them to increase it from 2 to 4 cores and 4 to 8GB RAM and now I have zero timeouts there. Maybe these builders are timing out because they have too little CPU time allocated to them in the hypervisor. @bradfitz <https://github.com/bradfitz> can correct me if I'm wrong, but IIRC, the power8 builders are 2 core, 4GB RAM only. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44422 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACH7BDGWKU6CUMSDJWWFK3TTAQDM5ANCNFSM4X4PGX5A> .

laboger · 2021-03-11T14:14:46Z

It's not clear to me whether the test is hung or just slow. Perhaps we should set a GO_TEST_TIMEOUT_SCALE on this builder and see if that helps?

@dmitshur I think we need to increase the GO_TEST_TIMEOUT_SCALE for all the linux ppc64* builders. I can make the change but don't know how those are tested.

dmitshur · 2021-03-11T15:23:47Z

@laboger Sounds good. GO_TEST_TIMEOUT_SCALE is a part of builder configuration, it doesn't need tests. See CL 158537 for a past example.

gopherbot · 2021-03-11T19:10:53Z

Change https://golang.org/cl/300870 mentions this issue: build: increase GO_TEST_TIMEOUT_SCALE for ppc64 builders

This sets the GO_TEST_TIMEOUT_SCALE=2 for ppc64 builders. The runtime tests have been timing out intermittently on ppc64le power8; it has happened on ppc64 and ppc64le power9 but very rarely. Updates golang/go#44422 Change-Id: I663f3f211a368a59e38fbff9ce43c925c6c7a209 Reviewed-on: https://go-review.googlesource.com/c/build/+/300870 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Trust: Lynn Boger <laboger@linux.vnet.ibm.com>

bcmills · 2021-04-28T13:05:09Z

This turns out not to be specific to the ppc64le builder. I think the runtime tests are just getting too slow in short mode in general.

(CC @jeremyfaller)

bcmills · 2021-05-07T13:45:20Z

This may be the same underlying issue as #45885.

bcmills added Builders NeedsInvestigation arch-ppc64x labels Feb 19, 2021

bcmills added this to the Backlog milestone Feb 19, 2021

bcmills changed the title ~~runtime: timeouts in buildTestProg on linux-ppc64le-buildlet~~ runtime: timeouts in buildTestProg Apr 28, 2021

bcmills removed arch-ppc64x Builders labels Apr 28, 2021

bcmills mentioned this issue Jun 1, 2021

runtime: frequent test timeouts on illumos-amd64 builder since March #46497

Open

golang / go Public

runtime: timeouts in buildTestProg #44422

runtime: timeouts in buildTestProg #44422

bcmills commented Feb 19, 2021

laboger commented Feb 23, 2021 •

edited

ceseo commented Feb 23, 2021

laboger commented Feb 24, 2021

laboger commented Mar 11, 2021 •

edited

dmitshur commented Mar 11, 2021

gopherbot commented Mar 11, 2021

bcmills commented Apr 28, 2021

bcmills commented May 7, 2021

golang / go Public

runtime: timeouts in buildTestProg #44422

runtime: timeouts in buildTestProg #44422

Comments

bcmills commented Feb 19, 2021

laboger commented Feb 23, 2021 • edited

ceseo commented Feb 23, 2021

laboger commented Feb 24, 2021

laboger commented Mar 11, 2021 • edited

dmitshur commented Mar 11, 2021

gopherbot commented Mar 11, 2021

bcmills commented Apr 28, 2021

bcmills commented May 7, 2021

laboger commented Feb 23, 2021 •

edited

laboger commented Mar 11, 2021 •

edited