x/tools: multiple tests timing out on darwin-amd64-longtest
#59634
Comments
gopherbot
added
the
Tools
bcmills
added
OS-Darwin
Builders
|
The current That gives a timeout of ~50m per test, which does seem rather long, but barring some easy way to speed up the builder maybe it's ok not to have great throughput for @golang/release, if we bump the scale factor higher are we going to be at risk of running into some other |
The main timeout that comes to mind is the 2 hour global VM timeout here (that an individual host can control for itself via A |
|
Talking with @prattmic, our theory is that maybe the guest VM is running out of RAM and burning a bunch of CPU time (and disk throughput) on swapping.
|
|
Change https://go.dev/cl/484746 mentions this issue: |
|
Change https://go.dev/cl/484747 mentions this issue: |
This reverts CL 461775. That CL failed to update cmd/runqemubuildlet, which is used for production VM execution. Thus it had no impact. The next CL will change memory, so just revert this for now rather than fixing to avoid changing two things at once. For golang/go#48945. For golang/go#59634. Change-Id: I9f1e26f2b3ffd6caca9683242a44597b7f1d8d58 Reviewed-on: https://go-review.googlesource.com/c/build/+/484746 Run-TryBot: Michael Pratt <mpratt@google.com> Auto-Submit: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Commit-Queue: Michael Pratt <mpratt@google.com>
Increase memory allocation from 4GB to 10GB. x/tools (and perhaps others) tests are heavily swapping with only 4GB of memory, hurting performance. 10GB seems to be a good tradeoff of more memory without too much risk of swapping the host. At 12GB, QEMU was sometimes using >16GB of memory, leaving the host at risk of swapping if both guests used lots of memory at once. For golang/go#59634. Change-Id: I3d11439266aa592697ecf83926632f954c45e84f Reviewed-on: https://go-review.googlesource.com/c/build/+/484747 Commit-Queue: Michael Pratt <mpratt@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Pratt <mpratt@google.com> Auto-Submit: Michael Pratt <mpratt@google.com>
|
https://build.golang.org/?repo=golang.org%2fx%2ftools shows two passing runs in a row now. I suspect that the |
|
For comparison, on (but that VM also has more and faster CPU, and perhaps a faster disk as well.) |
|
Related: we're having a lot of trouble with the darwin builders in #64473. Gopls's marker tests, which run many small gopls sessions (each invoking the Go command around 7 times), are almost 100x slower on the Darwin builders. By comparison, on the linux-amd64 builder they run in ~8s: https://build.golang.org/log/81422c4913748749a9e2fdfc40197a036bf57b4e These are our most critical tests, and we may have to skip them with Is there anything we can do to scale the darwin builders such that they are more representative of a typical development machine? |
|
Following up on #59634 (comment): we have since disabled these tests with -short on legacy builders. LUCI builders seem to have better and more stable performance. |
https://build.golang.org/log/56b688f3bdc981ffdeb03abab6d7519abbb782a6 shows numerous failures & timeouts for the
x/toolsrepo on the newdarwin-amd64-longtestbuilder (#35678).They all appear to have goroutines in the
[runnable]state, so perhaps we just need to bump theGO_TEST_TIMEOUT_SCALEeven higher for this builder..? (attn @prattmic)golang.org/x/tools/cmd/godocgolang.org/x/tools/go/loadergolang.org/x/tools/go/pointergolang.org/x/tools/go/ssaThe text was updated successfully, but these errors were encountered: