x/build: bring back "broke the build" notifications

[josharian]

I monitor the dev.ssa dashboard for build failures, with decreasing frequency these days. It'd be nice to bring back the automatic notification.

cc @bradfitz

[rsc]

Sure would be nice!

[bradfitz]

@rsc, how is this a Go1.6 issue? It's not released with Go.

In any case, it's not exactly clear how this should work. Should a flake make it fire? Do you need N failures in a row first? What if the N failures in a row are different? How do you detect they're different? Parsing the text?

The easiest thing is to ignore flakes and just say any transition from ok to failure emails.

I suppose the coordinator could poll http://build.golang.org/?mode=json occasionally and post to Gerrit (which emails), since the coordinator already can post to Gerrit. By asking for the data from build.golang.org it would also catch builders like old-style builders like netbsd and solaris, which report directly to build.golang.org.

I'd prefer if people just don't break the build, though, so I've been spending my builder time primarily on making trybots better.

I'm happy to review code to resurrect this, though. I'm not sure where it lives before, or if it even makes sense to reuse any of the code. @adg?

[adg]

The code is already there in the build dashboard, it's just broken and I
haven't had an opportunity to investigate why.

On 31 October 2015 at 03:25, Brad Fitzpatrick notifications@github.com
wrote:

@rsc https://github.com/rsc, how is this a Go1.6 issue? It's not
released with Go.

In any case, it's not exactly clear how this should work. Should a flake
make it fire? Do you need N failures in a row first? What if the N failures
in a row are different? How do you detect they're different? Parsing the
text?

The easiest thing is to ignore flakes and just say any transition from ok
to failure emails.

I suppose the coordinator could poll http://build.golang.org/?mode=json
occasionally and post to Gerrit (which emails), since the coordinator
already can post to Gerrit. By asking for the data from build.golang.org
it would also catch builders like old-style builders like netbsd and
solaris, which report directly to build.golang.org.

I'd prefer if people just don't break the build, though, so I've been
spending my builder time primarily on making trybots better.

I'm happy to review code to resurrect this, though. I'm not sure where it
lives before, or if it even makes sense to reuse any of the code. @adg
https://github.com/adg?

—
Reply to this email directly or view it on GitHub
#12509 (comment).

[rsc]

Would still like this, and it would have potentially sped the work on Go 1.6, but too late now and obviously not going to gate Go 1.6.

[josharian]

cc @andybons

[andybons]

Indeed this would be nice.

[FiloSottile]

This would be useful both for the dev branches and for the subrepos (so we are notified when master breaks x/crypto or x/crypto breaks on 1.9).

[bradfitz]

This could even be in cmd/gopherbot (which is easy to hack on locally), looking at the build.golang.org failure results and trying to not spam too early and waiting until it's confident (e.g. make.bash failing is a real failure, but a test failure might be a flake unless we saw in N times in a row).

Look at x/build/cmd/retrybuilds for code that accesses the build.golang.org failure results.

/cc @toothrot

[gopherbot]

Change https://golang.org/cl/208322 mentions this issue: app/appengine: delete old "broke the build" code