x/build: use `go tool dist test -k` on fast builders?

[minux]

Especially for fast builders, I suggest that we make then use the keepgoing
mode of go tool dist test so that even if a test fails, all the remaining tests
are still tested.

This will be especially useful if a flaky tests fails and then masks the real
problem of a given a CL because the affected tests are not run.

As an example, https://golang.org/cl/19455 changes the runtime, but
the openbsd builder fails a unrelated net tests, and the runtime tests are
not run so we don't actually know if the CL catches any problems on
OpenBSD/386.
http://build.golang.org/log/38e92c1c987f7a7eeb4b6bb418c0d95ed706ba80

I don't know if it's worthwhile to also use the -k for the trybots. I can see
arguments from both sides.

/cc @bradfitz @adg

[adg]

I'm in favor of -k for fast builders. Seems like a good way to get a sense of overall build health.

I don't think trybots should use -k. There should be some deterrent to having flaky builds, and having the trybots fail seems like a good impetus for fixing those flakes.

[bradfitz]

The builders also don't use "go tool dist test" by itself anyway. The builder coordinator has its own logic for test sharding and when to stop.

I'm not sure how useful this would be in practice. Usually the first failure is enough and the rest would be noise.

I also want to read errors quickly from the builders and not waste more time finding more errors (especially if they'll be useless).

But as long as it can be done without slowing things done, I'm not opposed to this.

I don't have the time to work on this at the moment, though.

[bradfitz]

A bit more discussion is in #36089, like whether this should only be on release branches or everywhere.

But this bug also discusses policy of when we'd do this.

For TryBots I'd want to fail fast. Or I at least want the comment on Gerrit (and email) as fast as possible, even if it keeps going after the failure comment (which implies parsing the output as it's still running).

So for simplicity I'd say just use -k (or equivalent) on post-submit builds and make trybots fail fast.

/cc @dmitshur @bcmills

[gopherbot]

Change https://golang.org/cl/211678 mentions this issue: cmd/coordinator: use 'go tool dist test -k' on release branches