x/build: freebsd-amd64-race builder is failing consistently on golang.org/x/tools repo #36444
The linux-amd64-race builder is currently configured with 15 GB RAM (via the n1-standard-4 machine type). CL 213557 has recently increased the RAM for windows-amd64-race builder to 14.4 GB in order to work around 7.2 GB not being sufficient, as described at #35186 (comment).
If there aren't objections, I plan to bump its up RAM to be more similar in size (without exceeding the current value of 15 GB RAM used both by Linux and now Windows -race builders) to see it helps with the test failures and issues like #34621.
The text was updated successfully, but these errors were encountered:
Start using n1-highcpu-8 machine type instead of n1-highcpu-4 for the freebsd-amd64-race builder. The freebsd-amd64-race builder has produced good test results for the x/tools repo for a long time, but by now it has started to consistently fail for reasons that seem connected to it having only 3.6 GB memory. The Windows race builders needed to be bumped from 7.2 GB to 14.4 GB to run successfully, so this change makes a small incremental step to bring freebsd-amd64-race closer in line with other builders. If memory-related problems continue to occur with 7.2 GB, the next step will be to go up to 14.4 GB. The freebsd-amd64-race builder is using an older version of FreeBSD. We may want to start using a newer one for running tests with -race, but that should be a separate change so we can see the results of this change without another confounding variable. Also update all FreeBSD builders to use https in buildletURLTmpl, because it's expected to work fine and will be more consistent. Updates golang/go#36444 Updates golang/go#34621 Updates golang/go#29252 Updates golang/go#33986 Change-Id: Idfcefd1c91bddc9f70ab23e02fcdca54fda9d1ac Reviewed-on: https://go-review.googlesource.com/c/build/+/214433 Run-TryBot: Carlos Amedee <firstname.lastname@example.org> TryBot-Result: Gobot Gobot <email@example.com> Reviewed-by: Carlos Amedee <firstname.lastname@example.org>
The freebsd-amd64-race builder no longer failing consistently on x/tools. However, there are still occasional failures. Need to see if they are connected to limited memory, and if so, consider going up to 15 GB on the FreeBSD race builder to match the memory of Linux and Windows race builders.
Edit: Many of the failures are due to data races being detected. Filed issue #36605.
The failures in https://build.golang.org/log/74eea9d200fd346319a2595c2f2b2ab3f25c2a07 appear to be due to timeouts rather than data races, although it is possible that a data race failure is being masked by failure to propagate logs somewhere higher up the stack.
I think the build results are reliable, keeping the specific issues mentioned above in mind.
I'll close this since the original issue is resolved, and we can follow up with more specific issues as needed.
Whoops, I misread the issue and missed the "x/tools" prefix, thinking it was about the main Go repository.
This is still a problem for x/tools:
I'll re-open and look into this more later.
This is a followup to CL 214433. Start using n1-highcpu-16 machine type instead of n1-highcpu-8 for the freebsd-amd64-race builder. Increasing the RAM from 3.6 GB to 7.2 GB has helped golang/go#36444 significantly: the builder stopped failing consistently on x/tools and resulted in many data races being uncovered in golang/go#36605. However, by now, it has started to fail consistently again. This time it seems to be due to low performance, causing the tests in golang.org/x/tools/internal/lsp/regtest package to fail due with "context deadline exceeded" errors. FreeBSD is one of the ports that stays visible when "show only first- class ports" is checked on build.golang.org. The other -race builders have all been upgraded to the n1-highcpu-16 machine type by now out of necessity. It seems fair to provide the FreeBSD port with an equal amount of resources, even if the increased memory isn't strictly required yet. Once this change is applied, if the failures persist, we can be more confident that the problem is due to the code or the port, rather than due to this -race builder having 2𝗑 less CPU and RAM resources compared to other -race builders. An alternative is to increase timeout for this builder type, but I'm opting to defer exploring that after equalizing the machine type. For golang/go#36444. For golang/go#34621. For golang/go#29252. For golang/go#33986. Change-Id: I41f149365128c7bc6f576c778ac07618acc04612 Reviewed-on: https://go-review.googlesource.com/c/build/+/227859 Reviewed-by: Alexander Rakoczy <email@example.com>