cmd/go: frequent test timeouts on dragonfly-amd64 builder #45215
Comments
|
Probably it is the setup. Each builder is running in a VM but the virtual disks are stored in hard disks, not SSDs. Also, we have to periodically cleanup the filesystem more often than in a standard setup, in which the cleanup occurs once a day. If they are causing too much trouble, I could probably move the VMs to another server. Let me know! |
|
Based on that information about the builder, it sounds like it's expected it to need more time to complete tests successfully, so increasing the test timeout (#45216) has a good chance of resolving this issue. |
This sets the GO_TEST_TIMEOUT_SCALE=2 for dragonfly builders in order to try to solve intermittent test failures. For golang/go#45215. For golang/go#34034. Fixes golang/go#45216.
|
Change https://golang.org/cl/315329 mentions this issue: |
This sets the GO_TEST_TIMEOUT_SCALE=2 for dragonfly builders in order to try to solve intermittent test failures. For golang/go#45215. For golang/go#34034. Fixes golang/go#45216. Change-Id: I81dae533c137ff1903bf6b1a52cf9752e8921b7c GitHub-Last-Rev: 9b4fb80 GitHub-Pull-Request: #35 Reviewed-on: https://go-review.googlesource.com/c/build/+/315329 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Trust: Carlos Amedee <carlos@golang.org>
|
This one can be closed? |
|
This builder is still running extremely slowly, and I suspect (but haven't confirmed) that a lot of the problem comes down to disk performance. We may need to increase the timeout scale factor further, but I wonder if there are other issues at play. For example, is it possible that the builder is memory-starved? Maybe we need to artificially lower GOMAXPROCS to avoid hitting swap while running the tests... |
|
We're going to try and have make.bash working so that the builders can be run in GCE, see: #23060 That might help us a bit with the builder slowness (hopefully). |
2021-03-24T03:16:14-4357f71/dragonfly-amd64
2021-03-23T23:09:33-87a3ac5/dragonfly-amd64
2021-03-23T01:21:24-d25476e/dragonfly-amd64
2021-03-19T17:31:59-4829031/dragonfly-amd64
2021-03-19T16:11:47-a937729/dragonfly-amd64
2021-03-18T21:27:21-d3ab6b5/dragonfly-amd64
2021-03-18T21:22:04-bdbba22/dragonfly-amd64
2021-03-18T17:14:39-f47fab9/dragonfly-amd64
2021-03-18T16:51:27-eaa1dde/dragonfly-amd64
2021-03-18T16:51:24-af4388a/dragonfly-amd64
2021-03-18T04:17:00-c2d6251/dragonfly-amd64
The timeouts don't seem to occur in any one particular test case, and no individual test case seems to be running for an inordinately long time. I think this builder is just slow, but I'm filing this issue in case there is a specific performance problem in
cmd/goor the test itself that we can address.I wonder if something about the
dragonflykernel or the builder configuration in particular makes the I/O syscalls used bycmd/goslower than on other platforms.CC @tuxillo @jayconrod @matloob
The text was updated successfully, but these errors were encountered: