Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: timeouts in buildTestProg #44422

Open
bcmills opened this issue Feb 19, 2021 · 8 comments
Open

runtime: timeouts in buildTestProg #44422

bcmills opened this issue Feb 19, 2021 · 8 comments

Comments

@laboger
Copy link
Contributor

@laboger laboger commented Feb 23, 2021

I've been looking at this problem on various systems. If I run 'go tool dist test go_test:runtime' by itself the time varies and sometimes on a power8 it times out, but never on a power9 because it is faster there. But one mysterious result is that if I set GOMAXPROCS to a smaller value, like 2, 4, or 8 then the time is small enough on the power8 so that it doesn't time out. I thought these VMs only had 2 processors each, which I thought was equivalent to running GOMAXPROCS=2. I can only get it to timeout if the number of processors is > 150.

@ceseo
Copy link
Contributor

@ceseo ceseo commented Feb 23, 2021

I don't know if this is related, but a while back I had a bunch of timeouts in my personal OSU power9 system. I asked them to increase it from 2 to 4 cores and 4 to 8GB RAM and now I have zero timeouts there. Maybe these builders are timing out because they have too little CPU time allocated to them in the hypervisor.

@bradfitz can correct me if I'm wrong, but IIRC, the power8 builders are 2 core, 4GB RAM only.

@laboger
Copy link
Contributor

@laboger laboger commented Feb 24, 2021

@laboger
Copy link
Contributor

@laboger laboger commented Mar 11, 2021

It's not clear to me whether the test is hung or just slow. Perhaps we should set a GO_TEST_TIMEOUT_SCALE on this builder and see if that helps?

@dmitshur I think we need to increase the GO_TEST_TIMEOUT_SCALE for all the linux ppc64* builders. I can make the change but don't know how those are tested.

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Mar 11, 2021

@laboger Sounds good. GO_TEST_TIMEOUT_SCALE is a part of builder configuration, it doesn't need tests. See CL 158537 for a past example.

@gopherbot
Copy link

@gopherbot gopherbot commented Mar 11, 2021

Change https://golang.org/cl/300870 mentions this issue: build: increase GO_TEST_TIMEOUT_SCALE for ppc64 builders

gopherbot pushed a commit to golang/build that referenced this issue Mar 16, 2021
This sets the GO_TEST_TIMEOUT_SCALE=2 for ppc64 builders. The
runtime tests have been timing out intermittently on ppc64le
power8; it has happened on ppc64 and ppc64le power9 but
very rarely.

Updates golang/go#44422

Change-Id: I663f3f211a368a59e38fbff9ce43c925c6c7a209
Reviewed-on: https://go-review.googlesource.com/c/build/+/300870
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Trust: Lynn Boger <laboger@linux.vnet.ibm.com>
@bcmills bcmills changed the title runtime: timeouts in buildTestProg on linux-ppc64le-buildlet runtime: timeouts in buildTestProg Apr 28, 2021
@bcmills
Copy link
Member Author

@bcmills bcmills commented May 7, 2021

This may be the same underlying issue as #45885.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants