If a build failure for a given platform P can be considered a release-blocker (such as #29221), then trybots should also run on P. Otherwise we have to rely on the actual build failure before we can fix the issue.
Example: In #29221, some newly added math tests failed on s390x, yet the trybots didn't notice.
If it's too expensive (too slow) to run the trybots for some platforms all the time, maybe we could consider doing it at least some time before any of the imminent release stages. For instance, if we are planning to cut a Beta or RC, we might want to consider starting to run the trybots a week before on all platforms where failures might block the release.
If we consider breaking the platform to be a release-blocking bug, then I agree it should be run on every trybot run triggered by a CL. That said, for builders that we can't spin up arbitrary instances of like the IBM Z-series, it could create a bottleneck for all runs across every CL.
One thing we could do (though it's not a trivial task) is create a submit queue that requires all builders across first-class ports to succeed and if they do, the patch gets submitted. So intermediate runs during review are a smaller subset that run quicker but the final submission would catch this sort of issue.
@bradfitz did we ever have email alerts when a builder failed? Is there precedent for adding them?
Of the first class ports (the ones that we define as being release blockers), the two we're missing as trybots are linux/arm and darwin/amd64. We used to have both of those but they were too unreliable.
Things we can improve that would satisfy this bug, probably in this order:
improve slowbot reporting (make trybot status URLs permanent, survive restarts)
start with making CLs that look like they're touching arm code or darwin code auto-invoke slowbots
make slowbots report trybot happiness once trybots are happy, even if the slow ones are still running
make slowbots time out after inability to get a builder for, say, an hour
add some global test caching (#28950) so we don't waste builder resources running tests on, say, doc changes