A bit more discussion is in #36089, like whether this should only be on release branches or everywhere.
But this bug also discusses policy of when we'd do this.
For TryBots I'd want to fail fast. Or I at least want the comment on Gerrit (and email) as fast as possible, even if it keeps going after the failure comment (which implies parsing the output as it's still running).
So for simplicity I'd say just use -k (or equivalent) on post-submit builds and make trybots fail fast.
Right now, on release branches, testing is stopped very soon after
encountering the first package with a failing test. For example:
FAIL cmd/go/internal/modfetch 17.181s
ok cmd/go/internal/modfetch/codehost 5.474s
2019/11/07 02:35:34 Failed: exit status 1
Ideally we should not have failing tests on release branches for long,
but that is not the current state. Until we reach that state, make
coordinator run 'go tool dist test' with -k flag so that it keeps
testing all remaining packages even if one fails, and report complete
test results in the log.
That way, a package that fails early doesn't make it completely
impossible to see if other package tests pass or fail.
Run-TryBot: Dmitri Shuralyov <firstname.lastname@example.org>
TryBot-Result: Gobot Gobot <email@example.com>
Reviewed-by: Alexander Rakoczy <firstname.lastname@example.org>
dist test -k doesn't really do anything from the coordinator.
The coordinator doesn't use all.bash/run.bash so it doesn't do just one go tool dist test execution. It instead does go tool dist list to find the list of dist test names, and then it executes them each itself (usually shared out over multiple buildlets, to speed up test execution).