Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools: frequent "out of memory" errors on windows-amd64-race builder #39196

Open
bcmills opened this issue May 21, 2020 · 5 comments
Open

x/tools: frequent "out of memory" errors on windows-amd64-race builder #39196

bcmills opened this issue May 21, 2020 · 5 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented May 21, 2020

x/tools often runs out of memory when testing golang.org/x/tools/go/packages, and occasionally when running other tests too.

We should either upgrade the builder (see also #33986, #33951) or reduce the memory footprint of the test (see also #36943, #14113, #32834).

2020-05-19T20:57:26-57a9e44/windows-amd64-race
2020-05-07T20:50:54-480da3e/windows-amd64-race
2020-05-05T02:31:15-26f46d2/windows-amd64-race
2020-05-01T15:50:19-2658dc0/windows-amd64-race
2020-04-30T22:11:53-f26c0dd/windows-amd64-race
2020-04-29T21:33:35-127c98b/windows-amd64-race
2020-04-28T21:10:48-dbf5ce1/windows-amd64-race
2020-04-28T18:55:08-e9a00ec/windows-amd64-race
2020-04-28T02:10:58-7ae4988/windows-amd64-race
2020-04-27T21:46:58-4697a28/windows-amd64-race
2020-04-27T21:30:09-6552487/windows-amd64-race
2020-04-27T20:59:12-352a540/windows-amd64-race
2020-04-27T20:15:23-e564293/windows-amd64-race
2020-04-27T18:59:06-e0d5eeb/windows-amd64-race
2020-04-27T16:39:59-9ea0146/windows-amd64-race
2020-04-27T15:30:19-a90b730/windows-amd64-race
2020-04-27T12:55:06-479cc23/windows-amd64-race
2020-04-26T10:28:38-f3a5411/windows-amd64-race
2020-04-24T19:57:22-3585060/windows-amd64-race
2020-04-23T20:53:58-59e7361/windows-amd64-race
2020-04-23T20:44:50-38a97e0/windows-amd64-race
2020-04-23T20:11:57-2723c5d/windows-amd64-race
2020-04-23T18:13:49-a466788/windows-amd64-race
2020-04-22T20:52:58-72e4a01/windows-amd64-race
2020-04-22T17:07:37-3d37a67/windows-amd64-race
2020-04-22T02:23:33-3d57cf2/windows-amd64-race
2020-04-22T01:32:14-504fe87/windows-amd64-race
2020-04-21T14:47:19-26dd2a5/windows-amd64-race
2020-04-21T13:43:21-4b6a508/windows-amd64-race
2020-04-21T13:42:58-2dc4334/windows-amd64-race
2020-04-21T04:27:24-cfa8b22/windows-amd64-race
2020-04-21T04:22:56-b4d5569/windows-amd64-race
2020-04-20T21:06:19-9f075f7/windows-amd64-race
2020-04-20T21:05:32-434f7a8/windows-amd64-race
2020-04-20T21:05:12-98173f2/windows-amd64-race
2020-04-20T21:04:52-f0ebba1/windows-amd64-race
2020-04-20T00:18:25-978e26b/windows-amd64-race
2020-04-17T14:00:56-c07e33e/windows-amd64-race
2020-04-16T21:44:02-fc95973/windows-amd64-race
2020-04-16T19:38:27-92fa1ff/windows-amd64-race
2020-04-16T04:05:50-9c19d0d/windows-amd64-race
2020-04-15T03:45:06-5d8e189/windows-amd64-race
2020-04-14T21:18:25-33e9372/windows-amd64-race
2020-04-14T21:18:03-ce2627f/windows-amd64-race
2020-04-14T20:55:25-6a72e37/windows-amd64-race
2020-04-14T13:15:30-0037cb7/windows-amd64-race
2020-04-14T03:22:29-332987a/windows-amd64-race
2020-04-14T00:10:08-ae52e4b/windows-amd64-race
2020-04-13T22:35:07-07bb9fb/windows-amd64-race
2020-04-13T16:19:37-250b213/windows-amd64-race
2020-04-13T01:58:12-1f08ef6/windows-amd64-race
2020-04-13T01:57:29-b854406/windows-amd64-race
2020-04-10T19:49:07-79a7a31/windows-amd64-race
2020-04-10T18:16:43-e3f0bd9/windows-amd64-race
2020-04-10T04:07:51-3bd2087/windows-amd64-race
2020-04-10T03:56:59-c08edad/windows-amd64-race
2020-04-09T21:04:53-700752c/windows-amd64-race
2019-05-20T18:16:09-776e170/windows-amd64-race

CC @dmitshur @toothrot @cagedmantis @ianthehat

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented May 21, 2020

The windows-amd64-race builder is already using n1-highcpu-16 machine type (16 vCPUs, 14.4 GB mem), same as the windows-amd64-longtest builder. So, there isn't an easy opportunity to increase the RAM of the -race builder to match that of -longtest builder.

One difference between them is that the -race builder is using Windows server 2008, while -longtest is using Windows server 2016. Would it be a good first step to update the -race builder to a newer version of Windows? Or is it considered a benefit that it's testing with an older version of Windows?

@bcmills
Copy link
Member Author

@bcmills bcmills commented May 21, 2020

I think it makes sense to upgrade the -race builder to the latest OS image, since it's really checking for bugs on the Go side anyway.

(I doubt that will fix this issue, but it's a fine first step to eliminate a variable.)

@gopherbot
Copy link

@gopherbot gopherbot commented May 27, 2020

Change https://golang.org/cl/235419 mentions this issue: dashboard: update windows-amd64-race builder to Windows Server 2016 image

gopherbot pushed a commit to golang/build that referenced this issue Jun 2, 2020
…mage

The purpose of the -race builder is to check for bugs on the Go side,
so use the newer available Windows version with the expectation that
there should be fewer unfixed bugs left in it.

Take this easy step to eliminate a variable for issue golang/go#39196.

Also delete host-windows-amd64-2008-big, since it's no longer used.

For golang/go#39196.

Change-Id: I9095a73ab21a151e5972219bccd5a16b249765b4
Reviewed-on: https://go-review.googlesource.com/c/build/+/235419
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@JohnCClarke
Copy link

@JohnCClarke JohnCClarke commented Jun 11, 2020

My team sees a similar error very frequently when running lots of tests sequentially from Makefiles. Curiously adding a few seconds delay between tests appears to improve the stability (although, without hard statistics we might be fooling ourselves). Simply rerunning the tests is often enough for them to succeed.

This is using go version go1.14.2 windows/amd64 on Microsoft Windows [Version 10.0.18363.836], and my impression is that the situation has become worse over time. That could be either or both of newer releases of golang or Windows. I wonder if a Windows expert knows more details of how Windows allocates memory and/or address space, and whether there is a better way for golang, or golang users, to interact with Windows.

This issue is a real nuisance for me and my team. We are very willing to help investigate and/or try potential fixes if anyone needs. Thanks!


Update at 2020-06-22: We have discovered that golang executables running out of memory is specific to executing our long "make" runs under powershell. Executing the same "make" runs under cmd.exe works correctly.

IWBN to understand the root cause though.

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Jul 6, 2020

I've looked at the last 2 weeks of results (this is after the builder change in CL 235419), and I'm not spotting OOM-related failures. There are frequent failures on windows-amd64-race builder (and other race builders), but many of all seem to be due to #39865.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants