Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/go/packages: test timeout on freebsd-amd64-race builder #34621

Open
bcmills opened this issue Sep 30, 2019 · 7 comments
Open

x/tools/go/packages: test timeout on freebsd-amd64-race builder #34621

bcmills opened this issue Sep 30, 2019 · 7 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 30, 2019

https://build.golang.org/log/10a77e60ed1952a8bbd937801dfd04d3b8aab6d1 shows a test timeout on the freebsd-amd64-race builder.

It may be that the test was deadlocked, or it may be that the builder just needs a longer scale factor for running the test.

It looks like the test got stuck running a subprocess, so if the problem is a deadlock, it might help to have the go/packages tests terminate subprocesses when the test runs out of time so that we can get goroutine dumps. (The tests for the go command do something similar here.)

CC @ianthehat @matloob @heschik @dmitshur @toothrot @bradfitz

@gopherbot gopherbot added this to the Unreleased milestone Sep 30, 2019
@gopherbot gopherbot added the Tools label Sep 30, 2019
@bcmills bcmills added Builders and removed Tools labels Sep 30, 2019
@gopherbot gopherbot added the Tools label Sep 30, 2019
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 30, 2019

Change https://golang.org/cl/204200 mentions this issue: go/packages: add a timeout for TestLoadImportsC

gopherbot pushed a commit to golang/tools that referenced this issue Oct 31, 2019
This will help debug golang/go#34621 by threading a context that's
canceled when the test times out. This will kill a go command that's
running when the test times out (the go command is already run
using CommandContext with the config's context).

Updates golang/go#34621

Change-Id: Iad373fd41d34395e817e6de50dd9e5842b2ef623
Reviewed-on: https://go-review.googlesource.com/c/tools/+/204200
Run-TryBot: Michael Matloob <matloob@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Nov 19, 2019

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Nov 19, 2019

This also masked an apparent regression in x/tools/gopls/test (#35691).

@matloob

This comment has been minimized.

Copy link
Contributor

@matloob matloob commented Dec 4, 2019

I'm not sure what we should do. Should we decrease the timeout?

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Dec 5, 2019

Hmm, that last log only showed a signal: killed for golang.org/x/tools/go/packages, and the actual timeout was for golang.org/x/tools/internal/lsp/cmd.

So probably golang.org/x/tools/internal/lsp/cmd needs some timeouts. The failure for golang.org/x/tools/go/packages may have been an OOM condition instead — the co-occurrence with a failure fo x/tools/internal/lsp/cmd suggests a possible connection to #33951.

@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Jan 7, 2020

The failure for golang.org/x/tools/go/packages may have been an OOM condition instead — the co-occurrence with a failure fo x/tools/internal/lsp/cmd suggests a possible connection to #33951.

I've just filed #36444 for bringing the RAM of freebsd race builder closer to that of Linux and Windows. It's possible making that change will help with this issue.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jan 13, 2020

Change https://golang.org/cl/214433 mentions this issue: dashboard: upsize freebsd-amd64-race builder to 7.2 GB RAM

gopherbot pushed a commit to golang/build that referenced this issue Jan 14, 2020
Start using n1-highcpu-8 machine type instead of n1-highcpu-4
for the freebsd-amd64-race builder.

The freebsd-amd64-race builder has produced good test results
for the x/tools repo for a long time, but by now it has started
to consistently fail for reasons that seem connected to it having
only 3.6 GB memory. The Windows race builders needed to be bumped
from 7.2 GB to 14.4 GB to run successfully, so this change makes
a small incremental step to bring freebsd-amd64-race closer in
line with other builders. If memory-related problems continue to
occur with 7.2 GB, the next step will be to go up to 14.4 GB.

The freebsd-amd64-race builder is using an older version of FreeBSD.
We may want to start using a newer one for running tests with -race,
but that should be a separate change so we can see the results of
this change without another confounding variable.

Also update all FreeBSD builders to use https in buildletURLTmpl,
because it's expected to work fine and will be more consistent.

Updates golang/go#36444
Updates golang/go#34621
Updates golang/go#29252
Updates golang/go#33986

Change-Id: Idfcefd1c91bddc9f70ab23e02fcdca54fda9d1ac
Reviewed-on: https://go-review.googlesource.com/c/build/+/214433
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.