Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools: frequent out-of-memory errors on linux-arm builder #32834

Open
bcmills opened this issue Jun 28, 2019 · 6 comments
Open

x/tools: frequent out-of-memory errors on linux-arm builder #32834

bcmills opened this issue Jun 28, 2019 · 6 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Jun 28, 2019

The tests under golang.org/x/tools almost always fail on the linux-arm builder.

Looking at the build dashboard, the most frequent failure mode (by far) seems to result in errors of the form fork/exec /workdir/go/bin/go: cannot allocate memory *os.PathError in the golang.org/x/tools/internal/lsp tests:

https://build.golang.org/log/5fb7985054d8b682b81438c385ff16e8325a3aa6
https://build.golang.org/log/e422caa59d35f5c76e097fdadb60407ee2807368
https://build.golang.org/log/3e3e538cf33e70880da37c2dd1c716b733b4b009
https://build.golang.org/log/739f069936cc47cc800b4d898c505e17a9d4a7e1

However, some of the other tests have also flaked with various resource-exhaustion errors:

A similar cannot allocate memory failure mode for golang.org/x/tools/go/internal/gcimporter:
https://build.golang.org/log/99a21d9e2e1064d08890439b43de987e0bfcdf52

fatal error: runtime: out of memory in golang.org/x/tools/go/ssa:
https://build.golang.org/log/de56b0ffe0e39b38b9ed449bd0cd382be72eda69

pthread_create failed: Resource temporarily unavailable and no space left on device:
https://build.golang.org/log/7151dca16e38a6f61051f032b2f6b713253052ac


I don't know exactly what the resource footprint of the linux-arm builder is (@andybons or @dmitshur might know?), but if the x/tools tests are expected to pass within that footprint we may need to dial back the test concurrency on that builder.

CC @ianthehat

@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Jun 28, 2019

I don't know exactly what the resource footprint of the linux-arm builder is

Our linux/arm builders are on Scaleway, see here. There are 50 builders and they're C1 BareMetal instances. Documentation at https://www.scaleway.com/en/bare-metal-instances/ says C1 instances have 2 GB of RAM, 4 ARMv7 cores.

2 GB of RAM certainly isn't a lot. The next step up, C2S, has 8 GB RAM, but also 64-bit CPUs instead of the 32-bit we need (not sure if that's a blocker or not).

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Aug 29, 2019

Change https://golang.org/cl/192332 mentions this issue: dashboard: skip x/tools on the linux-arm builder

@bradfitz

This comment has been minimized.

Copy link
Member

@bradfitz bradfitz commented Aug 29, 2019

The other possibility is to use go test -p 1 on the linux-arm builder.

That'd require a bit of coordinator changes.

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Aug 29, 2019

I'm not entirely sure that would even fix it: I don't know how much of the memory usage is due to parallelism vs. large test-cases, or how much of it ends up in long-running LSP servers.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Aug 29, 2019

Change https://golang.org/cl/192336 mentions this issue: go/packages,internal/lsp: skip memory-intensive tests on the linux-arm builder

gopherbot pushed a commit to golang/tools that referenced this issue Aug 29, 2019
…m builder

Updates golang/go#32834
Updates golang/go#30309

Change-Id: I75edb84776216523d00fadfb2e2d94ae08533c03
Reviewed-on: https://go-review.googlesource.com/c/tools/+/192336
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Aug 30, 2019

Change https://golang.org/cl/192578 mentions this issue: all: skip more memory-intensive tests on linux-arm

gopherbot pushed a commit to golang/tools that referenced this issue Aug 30, 2019
Updates golang/go#32834

Change-Id: I9844dc09d9a6eb2e79a0b28a1e69ed018bfa1bff
Reviewed-on: https://go-review.googlesource.com/c/tools/+/192578
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot gopherbot added the Tools label Sep 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.