Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
x/build: linux-arm builder can't finish an all.bash run when test sharding isn't used #40872
When I run a trybot on linux/arm, I get "out of memory" or "no space left on device" errors.
Is there anything we can do to fix this? Can we get more memory and/or disk space on these builders?
I could work on making cmd/compile/internal/ssa tests take less memory, perhaps. Not sure how much we could save.
Can you please include links to CLs where you've seen this? That way we'll have more information (e.g., which commit was being tested exactly, etc.). How often is this happening? Did it start recently?
The linux-arm builder is defined https://github.com/golang/build/blob/148ff27ab5b70970002d390c9e1da4b861f6da9f/dashboard/builders.go#L1736-L1756. They run on Scaleway (also see here), so adjusting resources will be limited to what's available there (we might already be maxed out; but need to look again to be more confident).
I see that linux-arm trybots are currently disabled because of other issues:
tryBot: nil, // Issue 22748, Issue 22749
Is this issue about that builder when requested via SlowBots or something else?
This happens when using gomote to run all.bash manually:
Sorry, I guess I'm using the term "trybot" to mean both the thing that tests CLs as well as manual gomotes. I mean the latter (except in the context of my second comment).
This is not exactly the same. #35628 is about trybot, this is about "when test sharding isn't used" e.g. manual gomote runs. The trybot one has weird STALE errors, whereas this one is OOMing or out of disk space.
About disk space, if I remember correctly, last time I looked, the machine actually has reasonably sizable disk space, but we're running on a very small partition.
The current linux-arm builder is known to have trouble when used as a SlowBot. Start warning about it when the builder is requested via the TRY= SlowBot UI. I've considered also removing or disabling the "arm" SlowBot alias, but that would make it easier to miss that there's an issue, since SlowBots don't warn about unknown builders: If you specify an unknown TRY= token, it'll just ignore it and won't report an error. We can consider making further changes as this situation evolves. The goal here is to start notifying about a known problem sooner. For golang/go#35628. For golang/go#40872. Change-Id: Ibc1205720c44ec4823c632c04fc2f887368258c1 Reviewed-on: https://go-review.googlesource.com/c/build/+/249420 Reviewed-by: Carlos Amedee <email@example.com> Reviewed-by: Alexander Rakoczy <firstname.lastname@example.org>