New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: builds waiting on resources shouldn't be in building state #10716

Open
bradfitz opened this Issue May 6, 2015 · 2 comments

Comments

Projects
None yet
5 participants
@bradfitz
Member

bradfitz commented May 6, 2015

/cc @crawshaw @adg

The reverse BuildletPool impl is cheating a bit or interacting weirdly with the rest of the coordinator: instead of blocking on returning a buildlet, it instead "starts" a bunch of builds even if there's not an idle iPhone available. And then each builds looks like it's in progress on build.golang.org (with happy blue gopher), but in reality they're all just stuck, for 4 hours (!?) here:

screen shot 2015-05-06 at 9 04 59 am

Arguably, the GCE pool impl might do this too, but it happens much less often so it's been easier to miss and ignore.

We should have make quota a more first-class concept within the coordinator and have different states for "waiting" vs "building". And when quota becomes available, we should prioritize which waiting build gets to go next. Right now the one who gets lucky and grabs a mutex gets to go next.

Then, once we have two states, we should make build.golang.org have two colors of gophers: gray for waiting, and blue for actively building.

@minux

This comment has been minimized.

Member

minux commented May 10, 2015

@ianlancetaylor ianlancetaylor added this to the Unreleased milestone Jun 3, 2015

@gopherbot

This comment has been minimized.

gopherbot commented Jun 11, 2015

CL https://golang.org/cl/10921 mentions this issue.

bradfitz added a commit to golang/build that referenced this issue Jun 11, 2015

cmd/coordinator: delay dashboard pinging until build starts
There were too many blue gophers too early on build.golang.org.
Wait until we have quota first (GCE quota or reverse buildlet pool
aquired).

Updates golang/go#10716

Change-Id: I076e89f319d68d55a61f3f8faf0242342c5ae825
Reviewed-on: https://go-review.googlesource.com/10921
Reviewed-by: Andrew Gerrand <adg@golang.org>

@rsc rsc added the Builders label Jun 11, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment