Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: OOM on linux-ppc64le-power10osu builder #58261

Closed
gopherbot opened this issue Feb 2, 2023 · 7 comments
Closed

x/build: OOM on linux-ppc64le-power10osu builder #58261

gopherbot opened this issue Feb 2, 2023 · 7 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@gopherbot
Copy link

gopherbot commented Feb 2, 2023

#!watchflakes
post <- builder == "linux-ppc64le-power10osu" && log ~ `signal: killed`

Issue created automatically to collect these failures.

Example (log):

# go run run.go -- rotate1.go
exit status 1
command-line-arguments: /workdir/go/pkg/tool/linux_ppc64le/compile: signal: killed

watchflakes

@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 2, 2023
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "rotate1.go" && test == ""
2023-01-31 16:54 linux-ppc64le-power10osu go@55a33d88 rotate1.go (log)
# go run run.go -- rotate1.go
exit status 1
command-line-arguments: /workdir/go/pkg/tool/linux_ppc64le/compile: signal: killed

watchflakes

@cherrymui cherrymui changed the title rotate1.go: unrecognized failures x/build: OOM on linux-ppc64le-power10osu builder Feb 2, 2023
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Feb 2, 2023
@gopherbot gopherbot added this to the Unreleased milestone Feb 2, 2023
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "linux-ppc64le-power10osu" && log ~ `signal: killed`
2023-01-31 16:53 linux-ppc64le-power10osu go@47e205c3 (log)
FAIL
2023/01/31 17:17:29 Failed: exit status 1
go tool dist: FAILED
2023-01-31 16:53 linux-ppc64le-power10osu go@43115ff0 (log)
FAIL
2023/01/31 17:17:50 Failed: exit status 1
go tool dist: FAILED
2023-02-01 21:30 linux-ppc64le-power10osu go@bd749504 chanlinear.go (log)
# go run run.go -- chanlinear.go
signal: killed
2023-02-01 21:30 linux-ppc64le-power10osu go@cda461bb index0.go (log)
# go run run.go -- index0.go
exit status 1
command-line-arguments: /workdir/go/pkg/tool/linux_ppc64le/compile: signal: killed

watchflakes

@bcmills
Copy link
Member

bcmills commented Feb 2, 2023

(attn @golang/ppc64)

@pmur
Copy link
Contributor

pmur commented Feb 2, 2023

Thanks. I suspect this was VM configuration issue which is hopefully resolved now.

Background, the VM was setup with too many vcpus (160), and only 30GB of RAM. At the time, OSU had some issues configuring the vcpu count. It's set appropriately now (24, as it needed to be a multiple of 8). Those unused cores may have caused excess RAM usage on the host.

@cherrymui
Copy link
Member

Thanks @pmur ! Sounds like we can close this for now. We can reopen if this happens again.

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "linux-ppc64le-power10osu" && log ~ `signal: killed`
2023-10-11 21:58 linux-ppc64le-power10osu text@6c97a165 go@31887586 (log)
FAIL
2023-10-11 21:58 linux-ppc64le-power10osu text@6c97a165 go@ea14b633 (log)
FAIL
2023-10-11 21:58 linux-ppc64le-power10osu text@6c97a165 go@3b303fa9 (log)
FAIL

watchflakes

@gopherbot gopherbot reopened this Nov 13, 2023
@pmur
Copy link
Contributor

pmur commented Nov 13, 2023

I killed any compile process running for more than 24 hours on the PPC64 builders due to #64067. The above are not related.

@pmur pmur closed this as completed Nov 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Status: Done
Development

No branches or pull requests

4 participants