Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: freebsd-amd64-race builder is failing consistently on golang.org/x/tools repo #36444

Closed
dmitshur opened this issue Jan 7, 2020 · 8 comments

Comments

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Jan 7, 2020

The linux-amd64-race builder is currently configured with 15 GB RAM (via the n1-standard-4 machine type). CL 213557 has recently increased the RAM for windows-amd64-race builder to 14.4 GB in order to work around 7.2 GB not being sufficient, as described at #35186 (comment).

The freebsd-amd64-race builder is currently configured to use the n1-highcpu-4 machine type, which has 3.6 GB RAM. The freebsd-amd64-race builder is failing quite often, which may be connected to insufficient memory:


signal: killed
FAIL	golang.org/x/tools/go/packages	217.792s
*** Test killed with quit: ran too long (10m0s).
FAIL	golang.org/x/tools/internal/imports	600.067s
panic: test timed out after 10m0s
[...]
FAIL	golang.org/x/tools/go/packages	611.592s
[...]
	package = golang.org/x/tools/internal/lsp/protocol
2020/01/07 18:44:29 Error:2020/01/07 18:44:29 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
2020/01/07 18:44:35 no dep handle: no metadata for golang.org/x/tools/internal/lsp/protocol
	package = golang.org/x/tools/internal/lsp/protocol
2020/01/07 18:44:35 Error:2020/01/07 18:44:35 no dep handle: no metadata for golang.org/x/tools/internal/lsp/protocol
	package = golang.org/x/tools/internal/lsp/protocol
2020/01/07 18:44:36 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
2020/01/07 18:44:36 Error:2020/01/07 18:44:36 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
2020/01/07 18:44:42 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
2020/01/07 18:44:43 Error:2020/01/07 18:44:42 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
--- FAIL: TestCommandLine (240.28s)
    --- FAIL: TestCommandLine/Modules (194.21s)
        --- FAIL: TestCommandLine/Modules/Diagnostics (159.73s)
            --- FAIL: TestCommandLine/Modules/Diagnostics/testy_test (151.73s)
                check.go:62: missing diagnostic "testy/testy_test.go:6:6: x declared but not used", map[]
FAIL
FAIL	golang.org/x/tools/internal/lsp/cmd	240.922s
2020/01/07 20:54:34 no dep handle: no metadata for nosuchpkg
	package = nosuchpkg
2020/01/07 20:54:51 no dep handle: no metadata for golang.org/x/tools/internal/lsp/protocol
	package = golang.org/x/tools/internal/lsp/protocol
2020/01/07 20:54:51 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
FAIL	golang.org/x/tools/internal/lsp/source	246.068s

If there aren't objections, I plan to bump its up RAM to be more similar in size (without exceeding the current value of 15 GB RAM used both by Linux and now Windows -race builders) to see it helps with the test failures and issues like #34621.

/cc @bradfitz @bcmills @matloob @toothrot

@gopherbot
Copy link

@gopherbot gopherbot commented Jan 13, 2020

Change https://golang.org/cl/214433 mentions this issue: dashboard: upsize freebsd-amd64-race builder to 7.2 GB RAM

@dmitshur dmitshur self-assigned this Jan 13, 2020
gopherbot pushed a commit to golang/build that referenced this issue Jan 14, 2020
Start using n1-highcpu-8 machine type instead of n1-highcpu-4
for the freebsd-amd64-race builder.

The freebsd-amd64-race builder has produced good test results
for the x/tools repo for a long time, but by now it has started
to consistently fail for reasons that seem connected to it having
only 3.6 GB memory. The Windows race builders needed to be bumped
from 7.2 GB to 14.4 GB to run successfully, so this change makes
a small incremental step to bring freebsd-amd64-race closer in
line with other builders. If memory-related problems continue to
occur with 7.2 GB, the next step will be to go up to 14.4 GB.

The freebsd-amd64-race builder is using an older version of FreeBSD.
We may want to start using a newer one for running tests with -race,
but that should be a separate change so we can see the results of
this change without another confounding variable.

Also update all FreeBSD builders to use https in buildletURLTmpl,
because it's expected to work fine and will be more consistent.

Updates golang/go#36444
Updates golang/go#34621
Updates golang/go#29252
Updates golang/go#33986

Change-Id: Idfcefd1c91bddc9f70ab23e02fcdca54fda9d1ac
Reviewed-on: https://go-review.googlesource.com/c/build/+/214433
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented Jan 14, 2020

CL 214433 was deployed this morning, and the freebsd-amd64-race builder has been passing on x/tools since then:

I'll give it more time to gather more data, and close this issue if the reliable build results continue.

@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented Jan 16, 2020

The freebsd-amd64-race builder no longer failing consistently on x/tools. However, there are still occasional failures. Need to see if they are connected to limited memory, and if so, consider going up to 15 GB on the FreeBSD race builder to match the memory of Linux and Windows race builders.

Edit: Many of the failures are due to data races being detected. Filed issue #36605.

@bcmills
Copy link
Member

@bcmills bcmills commented Feb 20, 2020

The failures in https://build.golang.org/log/74eea9d200fd346319a2595c2f2b2ab3f25c2a07 appear to be due to timeouts rather than data races, although it is possible that a data race failure is being masked by failure to propagate logs somewhere higher up the stack.

@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented Apr 7, 2020

I'll give it more time to gather more data, and close this issue if the reliable build results continue.

I think the build results are reliable, keeping the specific issues mentioned above in mind.

image

I'll close this since the original issue is resolved, and we can follow up with more specific issues as needed.

@dmitshur dmitshur closed this Apr 7, 2020
@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented Apr 7, 2020

Whoops, I misread the issue and missed the "x/tools" prefix, thinking it was about the main Go repository.

This is still a problem for x/tools:

https://build.golang.org/?repo=golang.org%2fx%2ftools

image

I'll re-open and look into this more later.

@dmitshur dmitshur reopened this Apr 7, 2020
@dmitshur dmitshur changed the title x/tools: freebsd-amd64-race builder is failing consistently x/build: freebsd-amd64-race builder is failing consistently on golang.org/x/tools repo Apr 7, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 10, 2020

Change https://golang.org/cl/227859 mentions this issue: dashboard: upsize freebsd-amd64-race builder to 16 vCPUs, 14.4 GB mem

gopherbot pushed a commit to golang/build that referenced this issue Apr 10, 2020
This is a followup to CL 214433. Start using n1-highcpu-16 machine type
instead of n1-highcpu-8 for the freebsd-amd64-race builder.

Increasing the RAM from 3.6 GB to 7.2 GB has helped golang/go#36444
significantly: the builder stopped failing consistently on x/tools
and resulted in many data races being uncovered in golang/go#36605.

However, by now, it has started to fail consistently again. This
time it seems to be due to low performance, causing the tests in
golang.org/x/tools/internal/lsp/regtest package to fail due with
"context deadline exceeded" errors.

FreeBSD is one of the ports that stays visible when "show only first-
class ports" is checked on build.golang.org. The other -race builders
have all been upgraded to the n1-highcpu-16 machine type by now out
of necessity.

It seems fair to provide the FreeBSD port with an equal amount of
resources, even if the increased memory isn't strictly required yet.
Once this change is applied, if the failures persist, we can be more
confident that the problem is due to the code or the port, rather
than due to this -race builder having 2𝗑 less CPU and RAM resources
compared to other -race builders.

An alternative is to increase timeout for this builder type, but I'm
opting to defer exploring that after equalizing the machine type.

For golang/go#36444.
For golang/go#34621.
For golang/go#29252.
For golang/go#33986.

Change-Id: I41f149365128c7bc6f576c778ac07618acc04612
Reviewed-on: https://go-review.googlesource.com/c/build/+/227859
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented Apr 10, 2020

CL 227859 has been deployed earlier today. The freebsd-amd64-race builder has started to pass on x/tools:

I'll consider this resolved. If something unexpected comes up, we can re-open this issue or file a new one.

@dmitshur dmitshur closed this Apr 10, 2020
@golang golang locked and limited conversation to collaborators Apr 10, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants