Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/coordinator: write proper scheduler #19178

Open
bradfitz opened this issue Feb 18, 2017 · 5 comments

Comments

@bradfitz
Copy link
Member

commented Feb 18, 2017

Currently if there are N builds waiting on a buildlet type (due to lack of machines or quota), the current implementation is a bunch of goroutines fighting over a mutex.

It's random who wins and gets the buildlet.

We should have them register interest and when a buildlet becomes available, pick the highest priority one.

Example priorities in order:

  • trybot run with a +2 already
  • trybot run
  • normal run of a build at tip of a release branch
  • normal run of a build at tip of master branch
  • normal run of a build at tip of dev branch
  • normal run of a build not at tip
  • idle flake detection (#19177)

etc.

/cc @danp @kevinburke

@bradfitz bradfitz added the Builders label Feb 18, 2017

@bradfitz bradfitz self-assigned this Feb 18, 2017

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Mar 13, 2017

And gomote access is in there somewhere too.

@gopherbot

This comment has been minimized.

Copy link

commented Apr 11, 2017

CL https://golang.org/cl/38306 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 12, 2017
cmd/coordinator: break up status into active vs pending builds
Currently all builds start and think they're running, but most are
just fighting over a mutex to grab a builder. That will be fixed, but
in the meantime it's nice to see what's actually working vs what's
waiting on e.g. arm5 hardware which won't be available for hours.

This is a baby step towards more monitoring. Currently this is just HTML
output, but the same data could be exported via JSON or something else later
for graphing.

Updates golang/go#19178 (add a buildlet scheduler)
Updates golang/go#15760 (monitor everything)

Change-Id: I36e16ea0919afe8023fe7fedd981f2e857f0d6df
Reviewed-on: https://go-review.googlesource.com/40397
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Apr 17, 2017

CL https://golang.org/cl/40397 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 21, 2017
cmd/coordinator: run benchmarks on try work
Benchmarks are treated as unit tests and distributed to the test
helpers, which allows them to fit in our 5m trybot budget.

Currently we only run the go1 and x/benchmarks. Running package
benchmarks is a TODO.

This feature is disabled by default, and is enabled by the
"farmer-run-bench" project attribute.

Updates golang/go#19178
Updates golang/go#19871

Change-Id: I9c3a14da60c3662e7e2cb4e71953060915cc4364
Reviewed-on: https://go-review.googlesource.com/38306
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Oct 20, 2017

Change https://golang.org/cl/70430 mentions this issue: cmd/coordinator: attempt to run newer work before older work on buildlets

gopherbot pushed a commit to golang/build that referenced this issue Oct 20, 2017
cmd/coordinator: attempt to run newer work before older work on build…
…lets

The problem is especially severe on the slow builders,
like darwin/arm64, which can sometimes not have run
any commit from the past 12 hours and still pick an old
commit for its next trial. It would be far better for it to
prefer new work.

It's possible that this new logic should be disabled for
some of the auto-scaling builders, but for now it seems
like we can try it for all of them and see if that's OK.

Update golang/go#19178.

Change-Id: I32cc67c0c2c84130b40b250675b40aadb4a0a681
Reviewed-on: https://go-review.googlesource.com/70430
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Sarah Adams <shadams@google.com>

@dmitshur dmitshur self-assigned this Aug 24, 2018

@gopherbot

This comment has been minimized.

Copy link

commented Aug 29, 2018

Change https://golang.org/cl/132076 mentions this issue: cmd/coordinator: start of a scheduler, not yet enabled

gopherbot pushed a commit to golang/build that referenced this issue Sep 26, 2018
cmd/coordinator: start of a scheduler, not yet enabled
Updates golang/go#19178

Change-Id: I24aa368df01a85259b53d6cfb08de7ab3a80e4fe
Reviewed-on: https://go-review.googlesource.com/132076
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.