Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools: frequent out-of-memory errors on linux-arm builder #32834

Open
bcmills opened this issue Jun 28, 2019 · 9 comments
Open

x/tools: frequent out-of-memory errors on linux-arm builder #32834

bcmills opened this issue Jun 28, 2019 · 9 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Testing An issue that has been verified to require only test changes, not just a test failure. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone
Unreleased

Comments

@bcmills
Copy link
Contributor

bcmills commented Jun 28, 2019

The tests under golang.org/x/tools almost always fail on the linux-arm builder.

Looking at the build dashboard, the most frequent failure mode (by far) seems to result in errors of the form fork/exec /workdir/go/bin/go: cannot allocate memory *os.PathError in the golang.org/x/tools/internal/lsp tests:

https://build.golang.org/log/5fb7985054d8b682b81438c385ff16e8325a3aa6
https://build.golang.org/log/e422caa59d35f5c76e097fdadb60407ee2807368
https://build.golang.org/log/3e3e538cf33e70880da37c2dd1c716b733b4b009
https://build.golang.org/log/739f069936cc47cc800b4d898c505e17a9d4a7e1

However, some of the other tests have also flaked with various resource-exhaustion errors:

A similar cannot allocate memory failure mode for golang.org/x/tools/go/internal/gcimporter:
https://build.golang.org/log/99a21d9e2e1064d08890439b43de987e0bfcdf52

fatal error: runtime: out of memory in golang.org/x/tools/go/ssa:
https://build.golang.org/log/de56b0ffe0e39b38b9ed449bd0cd382be72eda69

pthread_create failed: Resource temporarily unavailable and no space left on device:
https://build.golang.org/log/7151dca16e38a6f61051f032b2f6b713253052ac

I don't know exactly what the resource footprint of the linux-arm builder is (@andybons or @dmitshur might know?), but if the x/tools tests are expected to pass within that footprint we may need to dial back the test concurrency on that builder.

CC @ianthehat
@gopherbot gopherbot added this to the Unreleased milestone Jun 28, 2019
@bcmills bcmills added Builders x/build issues (builders, bots, dashboards) Testing An issue that has been verified to require only test changes, not just a test failure. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 28, 2019
@dmitshur
Copy link
Contributor

dmitshur commented Jun 28, 2019 •
edited
Loading

I don't know exactly what the resource footprint of the linux-arm builder is

Our linux/arm builders are on Scaleway, see here. There are 50 builders and they're C1 BareMetal instances. Documentation at https://www.scaleway.com/en/bare-metal-instances/ says C1 instances have 2 GB of RAM, 4 ARMv7 cores.

2 GB of RAM certainly isn't a lot. The next step up, C2S, has 8 GB RAM, but also 64-bit CPUs instead of the 32-bit we need (not sure if that's a blocker or not).

@bcmills bcmills mentioned this issue Jun 28, 2019
Open
@bcmills bcmills mentioned this issue Aug 29, 2019
Closed
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/192332 mentions this issue: dashboard: skip x/tools on the linux-arm builder

@bradfitz
Copy link
Contributor

The other possibility is to use go test -p 1 on the linux-arm builder.

That'd require a bit of coordinator changes.

@bcmills
Copy link
Contributor Author

bcmills commented Aug 29, 2019

I'm not entirely sure that would even fix it: I don't know how much of the memory usage is due to parallelism vs. large test-cases, or how much of it ends up in long-running LSP servers.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/192336 mentions this issue: go/packages,internal/lsp: skip memory-intensive tests on the linux-arm builder

gopherbot pushed a commit to golang/tools that referenced this issue Aug 29, 2019
go/packages,internal/lsp: skip memory-intensive tests on the linux-ar…
5d7988b 
…m builder

Updates golang/go#32834
Updates golang/go#30309

Change-Id: I75edb84776216523d00fadfb2e2d94ae08533c03
Reviewed-on: https://go-review.googlesource.com/c/tools/+/192336
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/192578 mentions this issue: all: skip more memory-intensive tests on linux-arm

gopherbot pushed a commit to golang/tools that referenced this issue Aug 30, 2019
all: skip more memory-intensive tests on linux-arm
311ec03 
Updates golang/go#32834

Change-Id: I9844dc09d9a6eb2e79a0b28a1e69ed018bfa1bff
Reviewed-on: https://go-review.googlesource.com/c/tools/+/192578
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Sep 12, 2019
@millerresearch
Copy link
Contributor

This issue applies (even more so) to the plan9-arm builders, which are mainly Raspberry Pi 3B+ boards with 1GB of RAM. I'll submit a CL to make the ExitIfSmallMachine test apply to plan9-arm, as it does to linux-arm.

@millerresearch millerresearch mentioned this issue May 6, 2020
Closed
@bcmills bcmills mentioned this issue May 21, 2020
Closed
@findleyr findleyr mentioned this issue Feb 12, 2021
Closed
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/291469 mentions this issue: gopls/internal/regtest: exit if small machine

gopherbot pushed a commit to golang/tools that referenced this issue Feb 12, 2021
@findleyr
gopls/internal/regtest: exit if small machine
ed2b1e9 
The issues we've experienced with linux-arm appear to simply be due to
memory constraints, which is already tracked in golang/go#32834 and
guarded by testenv.ExitIfSmallMachine. No issues on the larger
linux-arm-aws builder.

Remove our guard in favor of testenv.ExitIfSmallMachine.

Fixes golang/go#43355

Change-Id: Idea84e4fff13e13ab9a7eb3414c6133fff0f1e8e
Reviewed-on: https://go-review.googlesource.com/c/tools/+/291469
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/322409 mentions this issue: internal/testenv: update linux-arm to linux-arm-scaleway

gopherbot pushed a commit to golang/tools that referenced this issue May 25, 2021
@findleyr
internal/testenv: update linux-arm to linux-arm-scaleway
874765c 
CL 303230 renamed the linux-arm builder to linux-arm-scaleway, but the
x/tools ExitIfSmallMachine guard was not updated accordingly, resulting
in consistent failures for some tests.

For golang/go#45931
For golang/go#46183
Updates golang/go#32834

Change-Id: Ief5f17fc61cb38cf6b7cf63b6cd5e64f9d56261e
Reviewed-on: https://go-review.googlesource.com/c/tools/+/322409
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
rinchsan pushed a commit to rinchsan/gosimports that referenced this issue May 30, 2021
@findleyr @rinchsan
internal/testenv: update linux-arm to linux-arm-scaleway
ee7a465 
CL 303230 renamed the linux-arm builder to linux-arm-scaleway, but the
x/tools ExitIfSmallMachine guard was not updated accordingly, resulting
in consistent failures for some tests.

For golang/go#45931
For golang/go#46183
Updates golang/go#32834

Change-Id: Ief5f17fc61cb38cf6b7cf63b6cd5e64f9d56261e
Reviewed-on: https://go-review.googlesource.com/c/tools/+/322409
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
@bcmills bcmills mentioned this issue Dec 1, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Testing An issue that has been verified to require only test changes, not just a test failure. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@bradfitz @dmitshur @bcmills @gopherbot @millerresearch