x/build: freebsd-amd64-race logs for x/tools marked as failing, but do not include error messages #49347
I think the process may be being killed due to memory exhaustion.
According to https://cloud.google.com/compute/vm-instance-pricing,
Since the default GOMAXPROCS is based on the CPU count, and the default test parallelism is based on GOMAXPROCS, I suspect that the increased CPU-to-RAM ratio on the
@findleyr, @golang/release: should we address this by adjusting the host shape, or by adding some kind of additional limiting to the test, or explicitly setting GOMAXPROCS on this builder (which would possibly reduce its testing throughput), or something else entirely?
Comparing these two recent builds:
We can see that the failure is not finishing all x/tools tests -- the last test reported is go/loader.
If this is OOMing in go/packages (which we suspect may lead to this type of silent failure), then I can think of two possible sources:
Looking specifically at some of the go/packages tests, I am also curious about why the tests are so slow and memory intensive, based on what they are doing. e.g. TestLoadDifferentPatterns takes 3s, but that seems extremely slow based on my understanding of our type-checking speed and the scope of that test.
I think so — well spotted!
That is an interesting theory! But that change was merged in July, and the first of these failures started in October.
My leading theory is still machine shape (this builder has a very high CPU-to-RAM ratio), but either or both of those could be aggravating factors as well.
This builder is still failing multiple times per day on
@golang/release: could we try scaling this builder back to a more balanced configuration?
This reduces the freebsd-amd64-race builder to 4 vCPUs instead of 16, which should also reduce the default GOMAXPROCS and default -test.parallel settings on that builder. The theory is that by switching to a standard CPU-to-RAM ratio, we will avoid OOMing in memory-bound x/tools tests, which scale their parallelism by GOMAXPROCS rather than the machine's total RAM size. For golang/go#49347 Change-Id: I6960efc5636fef7d2e6197aee33212a7823f0970 Reviewed-on: https://go-review.googlesource.com/c/build/+/382058 Trust: Bryan Mills <email@example.com> Run-TryBot: Bryan Mills <firstname.lastname@example.org> Reviewed-by: Dmitri Shuralyov <email@example.com> TryBot-Result: Gopher Robot <firstname.lastname@example.org>