Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools: build failing on windows-amd64-longtest #35186

Closed
bcmills opened this issue Oct 26, 2019 · 5 comments
Closed

x/tools: build failing on windows-amd64-longtest #35186

bcmills opened this issue Oct 26, 2019 · 5 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 26, 2019

The windows-amd64-longtest builder is failing in numerous ways on x/tools.

Looks like some of the same out-of-memory issues as in #33951, but there may be other failures too.

https://build.golang.org/log/591874cbe6ad7d7b59ad1f9ef2b3d34e81af20a4

CC @ianthehat @matloob @stamblerre

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Dec 12, 2019

I believe this is a release-blocker via #11811.

(CC @cagedmantis @dmitshur @toothrot)

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jan 7, 2020

Change https://golang.org/cl/213557 mentions this issue: dashboard: upsize windows-amd64-{longtest,race} builders to 14.4 GB RAM

@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Jan 7, 2020

I've spun up a bunch of VMs/laptops with Windows 7/10 (WindowsBuild was very helpful), some with 8 GB RAM and one with 16 GB.

8 GB

On 8 GB configurations, I was able to get go test golang.org/x/tools/... to pass, but could not get go test -race golang.org/x/tools/... (with -race) to pass even once, despite multiple attempts and test caching helping out:

tools race oom 1

tools race oom 2

All 7-8~ GB RAM were used up during a large portion of the time.

16 GB

With 16 GB, go test -race golang.org/x/tools/... (with -race) passed on first try.

image

RAM usage still reached nearly all 15~16 GB RAM used up some of the time.


I've sent CL 213557 which bumps the RAM from 7.2 to 14.4 on windows-amd64-{longtest,race} builders. I think it's a reasonable next step to try, since some other builders already get 16 GB. /cc @bradfitz

gopherbot pushed a commit to golang/build that referenced this issue Jan 7, 2020
Start using n1-highcpu-16 machine type instead of n1-highcpu-8
for windows-amd64-longtest and windows-amd64-race builders.

The primary motivation of this change is to increase amount of
RAM available on the race builder from 7.2 GB to 14.4 GB.
CL 208500 has helped by going from 3.6 to 7.2, but empirically
8 GB is not enough for all tests to pass, while 16 GB can be.
We already use 16 GB for some other race and/or long builders.

Updates golang/go#35186
Updates golang/go#33951
Updates golang/go#33986

Change-Id: I8031a517e6e44ce5e5faba085e7c781a1fcab92e
Reviewed-on: https://go-review.googlesource.com/c/build/+/213557
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Jan 7, 2020

The aforementioned change has had an effect. Both windows-amd64-longtest and -race have been passing on x/tools since the builder change was deployed this morning. I'll keep watching a bit more, and then close this issue if the pattern continues.

There have been instances where x/tools has passed on all builders with tip and on release-branch.go1.13:

image

image

The freebsd-amd64-race builder still fails due to low memory occasionally, but that's outside of scope of this issue. I'll use issue #36444 to track increasing its memory to match Linux and Windows if needed.

@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Jan 13, 2020

This issue is resolved now, based on the current state of the windows-amd64-longtest column on https://build.golang.org/?repo=golang.org%2fx%2ftools:


(Newer windows-amd64-longtest builds are on top.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.