x/build: set up alerts for errors reported to stackdriver

[s-mang]

Currently we get a bunch of errors reported, eg:

9/4/17 7:53AM: buildID: B32739cd61, name: windows-amd64-2016, hostType: host-windows-amd64-2016, error: failed to get a buildlet: Error creating instance: &{Code:QUOTA_EXCEEDED Location: Message:Quota 'CPUS' exceeded. Limit: 500.0 ForceSendFields:[] NullFields:[]}

8/4/17 7:53AM: buildID: B059b7649c, name: openbsd-amd64-60, hostType: host-openbsd-amd64-60, error: failed to get a buildlet: Failed to create instance: googleapi: Error 403: Quota 'CPUS' exceeded. Limit: 500.0, quotaExceeded
at main.(*buildStatus).reportErr (coordinator.go:1951)

... etc

Probably not a pressing issue, as the coordinator's retry logic is at least enough to ensure the builds eventually run.

EDIT:
we need to figure out which errors are noise, and alert on the rest.

[s-mang]

Though perhaps we should think about paying for more cpus.

[bradfitz]

Yeah, that's not an alertable issue.

That's normal during spikes of load and doesn't affect users or builds.

The coordinator tries to do its own throttling and dead reckoning of quota, but there seems to be some lag of the reported quota from GCP which causes our internal usage accounting & usage semaphore to get off.

Though perhaps we should think about paying for more cpus.

It's just a simple ticket to bump the quota, but it won't really help. During a few commits, we will almost always have more work to do than CPUs, regardless of our CPU limit. It's fine to hit the limit for a few minutes sometimes.

Better would be to finish the scheduler: #19178

[s-mang]

sounds good.

[bradfitz]

I think you should keep this bug open to fix your logging/alerting.

You shouldn't get false alarms.

[s-mang]

ah, no alerts generated. just error logged.
actually, i haven't set any alerts at all. still trying to get a feel for what is normal and what is not.
ok, i'll modify the bug to address this directly (decide what to alert on and set up the alerts) and re-open. sound good?

we should do something similar for the # reverse buildlet alerting stuff. b/c it's just noise ATM.
i can tackle that today. cc @shantuo

[gopherbot]

Change https://golang.org/cl/53353 mentions this issue: cmd/buildlet: normalize macstadium host names for monitoring

[gopherbot]

Change https://golang.org/cl/96416 mentions this issue: devapp: start of status handler for monitoring