Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: trybots should include all platforms that can contribute release-blockers #29239

Open
griesemer opened this issue Dec 13, 2018 · 6 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@griesemer
Copy link
Contributor

griesemer commented Dec 13, 2018

If a build failure for a given platform P can be considered a release-blocker (such as #29221), then trybots should also run on P. Otherwise we have to rely on the actual build failure before we can fix the issue.

Example: In #29221, some newly added math tests failed on s390x, yet the trybots didn't notice.

If it's too expensive (too slow) to run the trybots for some platforms all the time, maybe we could consider doing it at least some time before any of the imminent release stages. For instance, if we are planning to cut a Beta or RC, we might want to consider starting to run the trybots a week before on all platforms where failures might block the release.

cc: @golang/osp-team
cc: @bradfitz
cc: @dmitshur
cc: @andybons

@griesemer griesemer added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Dec 13, 2018
@andybons andybons added this to the Unreleased milestone Dec 13, 2018
@andybons andybons changed the title trybot: should run on all platforms that can contribute release-blockers x/build: trybots should include all platforms that can contribute release-blockers Dec 13, 2018
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Dec 13, 2018
@andybons
Copy link
Member

If we consider breaking the platform to be a release-blocking bug, then I agree it should be run on every trybot run triggered by a CL. That said, for builders that we can't spin up arbitrary instances of like the IBM Z-series, it could create a bottleneck for all runs across every CL.

One thing we could do (though it's not a trivial task) is create a submit queue that requires all builders across first-class ports to succeed and if they do, the patch gets submitted. So intermediate runs during review are a smaller subset that run quicker but the final submission would catch this sort of issue.

@bradfitz did we ever have email alerts when a builder failed? Is there precedent for adding them?

@bradfitz
Copy link
Contributor

bradfitz commented Dec 14, 2018

@bradfitz did we ever have email alerts when a builder failed? Is there precedent for adding them?

Yes. That is #12509. (You even commented on it. :))

@bradfitz
Copy link
Contributor

A submit queue would also ensure that syntactically-but-not-semantically merge conflicts don't break builds.

@bcmills
Copy link
Contributor

bcmills commented Dec 14, 2018

for builders that we can't spin up arbitrary instances of like the IBM Z-series, it could create a bottleneck

We could batch up TryBot changes on those platforms: pile up all of the changes that need TryBot testing in a queue, and have a single TryBot run test all of the changes in the queue as a batch.

(The downside to that approach, of course, is a higher false-positive rate, or the need to break up the batch and retry failing tests individually in case of regressions.)

@bradfitz
Copy link
Contributor

TryBots are supposed to be fast, though.

If we want the SlowBots, that is more #24539 and ... another bug I can't find right now about letting users pick which trybots to run instead of the default set.

For submit queue, that is basically #12482 combined with #9858.

@bradfitz
Copy link
Contributor

Slowbots (#34501) are now live.

Of the first class ports (the ones that we define as being release blockers), the two we're missing as trybots are linux/arm and darwin/amd64. We used to have both of those but they were too unreliable.

Things we can improve that would satisfy this bug, probably in this order:

  • improve slowbot reporting (make trybot status URLs permanent, survive restarts)
  • start with making CLs that look like they're touching arm code or darwin code auto-invoke slowbots
  • make slowbots report trybot happiness once trybots are happy, even if the slow ones are still running
  • make slowbots time out after inability to get a builder for, say, an hour
  • add some global test caching (x/build/cmd/coordinator: use cmd/go's build caching #28950) so we don't waste builder resources running tests on, say, doc changes
  • include linux/arm and darwin/* as slowbots always

@bradfitz bradfitz added the NeedsFix The path to resolution is known, but the work has not been done. label Oct 21, 2019
@gopherbot gopherbot removed the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Oct 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

5 participants