runtime: frequent timeouts in {build,runBuilt,run}TestProg #44422

bcmills · 2021-02-19T14:37:34Z

It's not clear to me whether the test is hung or just slow. Perhaps we should set a GO_TEST_TIMEOUT_SCALE on this builder and see if that helps?

CC @golang/release @bradfitz @laboger @ceseo

The text was updated successfully, but these errors were encountered:

laboger · 2021-02-23T17:48:53Z

I've been looking at this problem on various systems. If I run 'go tool dist test go_test:runtime' by itself the time varies and sometimes on a power8 it times out, but never on a power9 because it is faster there. But one mysterious result is that if I set GOMAXPROCS to a smaller value, like 2, 4, or 8 then the time is small enough on the power8 so that it doesn't time out. I thought these VMs only had 2 processors each, which I thought was equivalent to running GOMAXPROCS=2. I can only get it to timeout if the number of processors is > 150.

ceseo · 2021-02-23T20:08:30Z

I don't know if this is related, but a while back I had a bunch of timeouts in my personal OSU power9 system. I asked them to increase it from 2 to 4 cores and 4 to 8GB RAM and now I have zero timeouts there. Maybe these builders are timing out because they have too little CPU time allocated to them in the hypervisor.

@bradfitz can correct me if I'm wrong, but IIRC, the power8 builders are 2 core, 4GB RAM only.

laboger · 2021-02-24T17:05:07Z

From golang.org/x/build/env/linux-ppc64le/osuosl/NOTES: * go-le-bionic-1: (20 GB RAM, 50 GB disk, 10 cores, POWER9) Linux go-le-bionic-1 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux * go-le-bionic-2: (20 GB RAM, 50 GB disk, 10 cores, POWER8) Linux go-le-bionic-2 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

…

On Tue, Feb 23, 2021 at 2:08 PM Carlos Eduardo Seo ***@***.***> wrote: I don't know if this is related, but a while back I had a bunch of timeouts in my personal OSU power9 system. I asked them to increase it from 2 to 4 cores and 4 to 8GB RAM and now I have zero timeouts there. Maybe these builders are timing out because they have too little CPU time allocated to them in the hypervisor. @bradfitz <https://github.com/bradfitz> can correct me if I'm wrong, but IIRC, the power8 builders are 2 core, 4GB RAM only. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44422 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACH7BDGWKU6CUMSDJWWFK3TTAQDM5ANCNFSM4X4PGX5A> .

laboger · 2021-03-11T14:14:46Z

It's not clear to me whether the test is hung or just slow. Perhaps we should set a GO_TEST_TIMEOUT_SCALE on this builder and see if that helps?

@dmitshur I think we need to increase the GO_TEST_TIMEOUT_SCALE for all the linux ppc64* builders. I can make the change but don't know how those are tested.

dmitshur · 2021-03-11T15:23:47Z

@laboger Sounds good. GO_TEST_TIMEOUT_SCALE is a part of builder configuration, it doesn't need tests. See CL 158537 for a past example.

gopherbot · 2021-03-11T19:10:53Z

Change https://golang.org/cl/300870 mentions this issue: build: increase GO_TEST_TIMEOUT_SCALE for ppc64 builders

This sets the GO_TEST_TIMEOUT_SCALE=2 for ppc64 builders. The runtime tests have been timing out intermittently on ppc64le power8; it has happened on ppc64 and ppc64le power9 but very rarely. Updates golang/go#44422 Change-Id: I663f3f211a368a59e38fbff9ce43c925c6c7a209 Reviewed-on: https://go-review.googlesource.com/c/build/+/300870 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Trust: Lynn Boger <laboger@linux.vnet.ibm.com>

bcmills · 2021-04-28T13:05:09Z

This turns out not to be specific to the ppc64le builder. I think the runtime tests are just getting too slow in short mode in general.

(CC @jeremyfaller)

bcmills · 2021-05-07T13:45:20Z

This may be the same underlying issue as #45885.

bcmills · 2021-11-08T20:50:57Z

greplogs --dashboard -md -l -e '(?ms)test timed out.*runtime_test\.(?:run|runBuil[dt]|build)TestProg.*FAIL\s+runtime' --since=2021-05-01

bcmills · 2021-11-08T20:56:54Z

This is affecting at least five different builders, with many failures per month.

I think that makes it a release-blocker via #11811. All of the builders need to be passing tests reliably — if that means skipping more tests on non-longtest builders, we may need to prioritize which tests to run, or perhaps move some of the slower ones out to some package that won't also be run by users as part of go test all in their own modules. (Compare #48770.)

bcmills · 2021-11-16T15:46:13Z

Looking at the solaris-amd64-oraclerel timeouts, many of the process wait times are quite long (1–2 minutes).

That makes me wonder whether these timeouts are actually deadlocks — perhaps related to the ones observed in #48789?

bcmills · 2021-11-16T15:49:57Z

#49614 also shows an apparent deadlock with processes blocked in runtime.syscall_wait4. (It may be a duplicate of this issue.)

ianlancetaylor · 2021-11-17T03:03:36Z

It's interesting to note that in all the recent Solaris failures the runtime test times out while building a test with build flags, either from TestCgoExternalThreadSignal (building testprogcgo with -tags=threadprof) or TestCheckPtr (building testprog with -gcflags=all=-d=checkptr=1). The buildTestProg function serializes all builds using a mutex. It does a separate build for each binary (testprog, testprogcgo, ....) and the additional arguments being passed to go build, typically none but sometimes, as in these cases, present. Building testprogcgo naturally requires invoking the C compiler a few times, and also the C linker. So overall these extra builds could definitely slow down the test. Still hard to see how it takes as long as it does, though.

gopherbot · 2021-11-17T06:22:27Z

Change https://golang.org/cl/364654 mentions this issue: runtime: don't serialize all builds in test

bcmills · 2021-11-17T16:22:12Z

in all the recent Solaris failures the runtime test times out while building a test with build flags

That is a strong pattern but not universal. The stack traces for the failures listed in #49614 seem to be at a point where the test is running the compiled program (and has been for over a minute), not waiting on a build process.

Permit a test whose program is already built to run immediately, rather than waiting for another test to complete its build. For #44422 Change-Id: I2d1b35d055ee4c4251f4caef3b52dccc82b71a1b Reviewed-on: https://go-review.googlesource.com/c/go/+/364654 Trust: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org>

ianlancetaylor · 2021-11-17T18:18:58Z

After CL 364654 the time required to run the runtime tests on a Solaris system I have access to drops from 70 seconds to 51 seconds. CL 364755, if submitted, will drop it farther. I don't know if timeouts are the problem here, but they may be part of it. But there may also be a different problem.

bcmills · 2021-11-17T18:21:06Z

Seems like a good start, at least. We can watch the builders for a while and see whether it is empirically fixed.

bcmills · 2021-11-19T16:35:39Z

The last timeout in these functions was before the aforementioned changes, so I'm provisionally marking this as fixed.
(The reduction in these timeouts did draw my attention to a different runtime timeout, which I have filed as #49680.)

greplogs --dashboard -md -l -e '(?ms)test timed out.*runtime_test\.(?:run|runBuilt|build)TestProg.*FAIL\s+runtime' --since=2021-11-17

2021-11-17T18:14:41-0981724/openbsd-386-68

bcmills · 2021-11-23T16:22:54Z

Nope, still timing out occasionally:

greplogs --dashboard -md -l -e '(?ms)test timed out.*runtime_test\.(?:run|runBuilt|build)TestProg.*FAIL\s+runtime' --since=2021-11-20

2021-11-22T20:34:40-f13fcd9/solaris-amd64-oraclerel

ianlancetaylor · 2021-11-23T19:24:46Z

In the 2021-11-22 timeout the only tests still running are two waiting for go build -tags=threadprof testprogcgo to complete.

ianlancetaylor · 2021-11-23T20:21:21Z

On a Solaris system I have access to, clearing the cache and running go test -test.short -test.v -test.count=1 runtime takes 38.5 seconds. The last two tests to complete are TestCgoExternalThreadSIGPROF and TestCgoExternalThreadSignal which both take about 13 seconds. One test is building the test program, and the other is waiting for that build to complete (they both build the same test program). Actually running the test program takes a fraction of a second.

This is consistent with the builder failure just being slow and running out of time, especially since cmd/dist will be running other tests in parallel.

bcmills · 2021-12-06T20:02:07Z

greplogs --dashboard -md -l -e '(?ms)test timed out.*runtime_test\.(?:run|runBuilt|build)TestProg.*FAIL\s+runtime' --since=2021-11-23

2021-12-02T22:06:27-06dbf61/solaris-amd64-oraclerel
2021-12-02T20:45:47-3c6295d/openbsd-386-68
2021-11-29T19:24:34-68da368/solaris-amd64-oraclerel
2021-11-24T15:57:53-14f2b2a/openbsd-386-68
2021-11-24T15:55:56-465b402/openbsd-386-68
2021-11-24T15:55:56-465b402/solaris-amd64-oraclerel

All of these timeouts also appear to be actively compiling in TestCgoExternalThread.*.

Perhaps some of these slower tests should be skipped in short mode?

aclements · 2022-01-05T17:13:07Z

Perhaps some of these slower tests should be skipped in short mode?

Since we don't have a long builder for these, this seems like it would lose important test coverage.

or perhaps move some of the slower ones out to some package that won't also be run by users as part of go test all in their own modules.

This seems feasible. As you pointed out, this would give it its own timeout. It would require some nontrivial refactoring to pull out the testprog infrastructure, but we could do that.

bcmills · 2022-01-07T15:07:29Z

greplogs --dashboard -md -l -e '(?ms)test timed out.*runtime_test\.(?:run|runBuilt|build)TestProg.*FAIL\s+runtime' --since=2021-12-07

2022-01-06T23:39:43-042548b/solaris-amd64-oraclerel (building TestFakeTime)
2022-01-06T15:00:16-da7891f/netbsd-386-9_0-n1 (running TestCgoExecSignalMask)
2022-01-03T23:45:06-a893d0f/openbsd-386-68 (building TestCgoExternalThreadSIGPROF)
2021-12-22T23:31:40-d2ce939/solaris-amd64-oraclerel (building TestCgoExternalThreadSignal)
2021-12-16T21:02:37-ea26ce7/netbsd-arm64-bsiegert (building TestFakeTime)
2021-12-13T19:40:29-7bdbc73/openbsd-386-68 (building TestCgoExternalThreadSIGPROF)
2021-12-12T14:36:17-6b89773/solaris-amd64-oraclerel (building TestCgoExternalThreadSIGPROF)

aclements · 2022-01-18T21:41:56Z

A much simpler thing we could do is to increase the timeout for the runtime test. We already do this for the cmd/go tests [1] and I don't think that's any more artificial than factoring the cgo prog tests out into their own test.

[1] Or, at least, we try to. The code is weird.

bcmills · 2022-01-18T21:49:46Z

If increasing the timeout fixes the builder flakiness, I think it's a reasonable approach — Go users who run go test all should in general expect to either set a very generous timeout or scale it to their particular running environment.

That said, if the problem is just the timeout length, should we raise the scale factors in x/build/dashboard/builders.go instead of just for the specific test?

(It would be nice if we could eliminate the cmd/go special case in that code too, but that seems like a separate issue...)

ianlancetaylor · 2022-01-19T02:42:44Z

The timeouts are from building a variant of threadprogcgo with -tags=threadprof. We don't need that variant. I'll send a CL to remove it. That will remove the extra build time and should keep the tests from timing out. I hope.

gopherbot · 2022-01-19T02:48:32Z

Change https://golang.org/cl/379294 mentions this issue: runtime: remove -tags=threadprof in tests

Use an enviroment variable rather than a build tag to control starting a busy loop thread when testprogcgo starts. This lets us skip another build that invokes the C compiler and linker, which should avoid timeouts running the runtime tests. Fixes golang#44422 Change-Id: I516668d71a373da311d844990236566ff63e6d72 Reviewed-on: https://go-review.googlesource.com/c/go/+/379294 Trust: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>

bcmills added Builders NeedsInvestigation arch-ppc64x labels Feb 19, 2021

bcmills added this to the Backlog milestone Feb 19, 2021

bcmills changed the title ~~runtime: timeouts in buildTestProg on linux-ppc64le-buildlet~~ runtime: timeouts in buildTestProg Apr 28, 2021

bcmills removed arch-ppc64x Builders labels Apr 28, 2021

bcmills mentioned this issue Jun 1, 2021

runtime: frequent test timeouts on illumos-amd64 builder since March #46497

Closed

bcmills changed the title ~~runtime: timeouts in buildTestProg~~ runtime: frequent timeouts in {build,runBuild,run}TestProg Nov 8, 2021

bcmills added the release-blocker label Nov 8, 2021

bcmills removed this from the Backlog milestone Nov 8, 2021

bcmills added this to the Go1.18 milestone Nov 8, 2021

bcmills added okay-after-beta1 Testing labels Nov 8, 2021

bcmills removed the Testing label Nov 16, 2021

bcmills mentioned this issue Nov 16, 2021

runtime: apparent deadlock in TestFakeTime on solaris-amd64-oraclerel starting 2021-11-05 #49614

Closed

bcmills added the WaitingForInfo label Nov 17, 2021

bcmills changed the title ~~runtime: frequent timeouts in {build,runBuild,run}TestProg~~ runtime: frequent timeouts in {build,runBuilt,run}TestProg Nov 19, 2021

bcmills closed this as completed Nov 19, 2021

bcmills removed the WaitingForInfo label Nov 19, 2021

bcmills reopened this Nov 23, 2021

cherrymui removed the okay-after-beta1 label Dec 14, 2021

gopherbot closed this as completed in 2c2e081 Jan 20, 2022

runtime: frequent timeouts in {build,runBuilt,run}TestProg #44422

runtime: frequent timeouts in {build,runBuilt,run}TestProg #44422

bcmills commented Feb 19, 2021 •

edited

laboger commented Feb 23, 2021 •

edited

ceseo commented Feb 23, 2021

laboger commented Feb 24, 2021

laboger commented Mar 11, 2021 •

edited

dmitshur commented Mar 11, 2021

gopherbot commented Mar 11, 2021

bcmills commented Apr 28, 2021 •

edited

bcmills commented May 7, 2021

bcmills commented Nov 8, 2021

bcmills commented Nov 8, 2021

bcmills commented Nov 16, 2021

bcmills commented Nov 16, 2021

ianlancetaylor commented Nov 17, 2021 •

edited

gopherbot commented Nov 17, 2021

bcmills commented Nov 17, 2021

ianlancetaylor commented Nov 17, 2021 •

edited

bcmills commented Nov 17, 2021

bcmills commented Nov 19, 2021

bcmills commented Nov 23, 2021

ianlancetaylor commented Nov 23, 2021 •

edited

ianlancetaylor commented Nov 23, 2021

bcmills commented Dec 6, 2021

aclements commented Jan 5, 2022

bcmills commented Jan 7, 2022

aclements commented Jan 18, 2022

bcmills commented Jan 18, 2022

ianlancetaylor commented Jan 19, 2022

gopherbot commented Jan 19, 2022

runtime: frequent timeouts in {build,runBuilt,run}TestProg #44422

runtime: frequent timeouts in {build,runBuilt,run}TestProg #44422

Comments

bcmills commented Feb 19, 2021 • edited

laboger commented Feb 23, 2021 • edited

ceseo commented Feb 23, 2021

laboger commented Feb 24, 2021

laboger commented Mar 11, 2021 • edited

dmitshur commented Mar 11, 2021

gopherbot commented Mar 11, 2021

bcmills commented Apr 28, 2021 • edited

bcmills commented May 7, 2021

bcmills commented Nov 8, 2021

bcmills commented Nov 8, 2021

bcmills commented Nov 16, 2021

bcmills commented Nov 16, 2021

ianlancetaylor commented Nov 17, 2021 • edited

gopherbot commented Nov 17, 2021

bcmills commented Nov 17, 2021

ianlancetaylor commented Nov 17, 2021 • edited

bcmills commented Nov 17, 2021

bcmills commented Nov 19, 2021

bcmills commented Nov 23, 2021

ianlancetaylor commented Nov 23, 2021 • edited

ianlancetaylor commented Nov 23, 2021

bcmills commented Dec 6, 2021

aclements commented Jan 5, 2022

bcmills commented Jan 7, 2022

aclements commented Jan 18, 2022

bcmills commented Jan 18, 2022

ianlancetaylor commented Jan 19, 2022

gopherbot commented Jan 19, 2022

bcmills commented Feb 19, 2021 •

edited

laboger commented Feb 23, 2021 •

edited

laboger commented Mar 11, 2021 •

edited

bcmills commented Apr 28, 2021 •

edited

ianlancetaylor commented Nov 17, 2021 •

edited

ianlancetaylor commented Nov 17, 2021 •

edited

ianlancetaylor commented Nov 23, 2021 •

edited