x/build: use go tool dist test -k on fast builders?
#14305
Comments
minux
changed the title
go tool dist test -k on fast builders?go tool dist test -k on fast builders?
|
I'm in favor of I don't think trybots should use |
|
The builders also don't use "go tool dist test" by itself anyway. The builder coordinator has its own logic for test sharding and when to stop. I'm not sure how useful this would be in practice. Usually the first failure is enough and the rest would be noise. I also want to read errors quickly from the builders and not waste more time finding more errors (especially if they'll be useless). But as long as it can be done without slowing things done, I'm not opposed to this. I don't have the time to work on this at the moment, though. |
bradfitz
changed the title
go tool dist test -k on fast builders?go tool dist test -k on fast builders?
|
A bit more discussion is in #36089, like whether this should only be on release branches or everywhere. But this bug also discusses policy of when we'd do this. For TryBots I'd want to fail fast. Or I at least want the comment on Gerrit (and email) as fast as possible, even if it keeps going after the failure comment (which implies parsing the output as it's still running). So for simplicity I'd say just use |
|
Change https://golang.org/cl/211678 mentions this issue: |
Right now, on release branches, testing is stopped very soon after encountering the first package with a failing test. For example: [...] FAIL cmd/go/internal/modfetch 17.181s ok cmd/go/internal/modfetch/codehost 5.474s FAIL 2019/11/07 02:35:34 Failed: exit status 1 Ideally we should not have failing tests on release branches for long, but that is not the current state. Until we reach that state, make coordinator run 'go tool dist test' with -k flag so that it keeps testing all remaining packages even if one fails, and report complete test results in the log. That way, a package that fails early doesn't make it completely impossible to see if other package tests pass or fail. Updates golang/go#14305 Updates golang/go#36181 Change-Id: Ia674005ae45c6b9dbffa6f5b56d60b7ed38f6b34 Reviewed-on: https://go-review.googlesource.com/c/build/+/211678 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
|
Based on the latest commit results from https://build.golang.org/?branch=release-branch.go1.13, the change in CL 211678 was not sufficient to have an effect on release branches of the main Go repository. It still stopped short. E.g., here's https://build.golang.org/log/060b17127f4e1e7bc35e12e00058274c77a06d9c Something more needs to be done for it to have an effect. |
|
The coordinator doesn't use all.bash/run.bash so it doesn't do just one |
Especially for fast builders, I suggest that we make then use the keepgoing
mode of
go tool dist testso that even if a test fails, all the remaining testsare still tested.
This will be especially useful if a flaky tests fails and then masks the real
problem of a given a CL because the affected tests are not run.
As an example, https://golang.org/cl/19455 changes the runtime, but
the openbsd builder fails a unrelated net tests, and the runtime tests are
not run so we don't actually know if the CL catches any problems on
OpenBSD/386.
http://build.golang.org/log/38e92c1c987f7a7eeb4b6bb418c0d95ed706ba80
I don't know if it's worthwhile to also use the -k for the trybots. I can see
arguments from both sides.
/cc @bradfitz @adg
The text was updated successfully, but these errors were encountered: