Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: report failures fast but still make all test results available #36181

Open
dmitshur opened this issue Dec 17, 2019 · 4 comments
Open

x/build: report failures fast but still make all test results available #36181

dmitshur opened this issue Dec 17, 2019 · 4 comments

Comments

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Dec 17, 2019

(This feature request/enhancement is related to issue #34119, the goal is to improve developer experience at the expense of using more computational resources.)

In #14305 (comment), there is discussion of whether to use -k on which branches, and on post-submit only vs trybots too. There is a trade-off in that the flag improves information that a failed build reports, but makes the test failure get reported more slowly.

A more ideal (for user developer experience) solution is to report a failure fast, as soon as the first test fails, but keep going and make all test results available for later viewing by interested parties.

See the fast-finish builds feature of Travis CI for some related precedent.

/cc @bradfitz @bcmills @toothrot @cagedmantis

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 17, 2019

Change https://golang.org/cl/211678 mentions this issue: cmd/coordinator: use 'go tool dist test -k' on release branches

gopherbot pushed a commit to golang/build that referenced this issue Dec 19, 2019
Right now, on release branches, testing is stopped very soon after
encountering the first package with a failing test. For example:

	[...]
	FAIL	cmd/go/internal/modfetch	17.181s
	ok  	cmd/go/internal/modfetch/codehost	5.474s
	FAIL
	2019/11/07 02:35:34 Failed: exit status 1

Ideally we should not have failing tests on release branches for long,
but that is not the current state. Until we reach that state, make
coordinator run 'go tool dist test' with -k flag so that it keeps
testing all remaining packages even if one fails, and report complete
test results in the log.

That way, a package that fails early doesn't make it completely
impossible to see if other package tests pass or fail.

Updates golang/go#14305
Updates golang/go#36181

Change-Id: Ia674005ae45c6b9dbffa6f5b56d60b7ed38f6b34
Reviewed-on: https://go-review.googlesource.com/c/build/+/211678
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@dmitshur
Copy link
Member Author

@dmitshur dmitshur commented Mar 17, 2020

/cc @findleyr

@gopherbot
Copy link

@gopherbot gopherbot commented Mar 17, 2020

Change https://golang.org/cl/223780 mentions this issue: [release-branch.go1.13] cmd/doc: skip failing TestDotSlashLookup on Windows

@gopherbot
Copy link

@gopherbot gopherbot commented Mar 17, 2020

Change https://golang.org/cl/223782 mentions this issue: [release-branch.go1.13] cmd/go: fix and skip known Windows test failures

gopherbot pushed a commit that referenced this issue Mar 31, 2020
…indows

This test was fixed by changing cmd/doc behavior in CL 204442.

Backporting that non-test code change is unlikely to be appropriate
this late in Go 1.13 release cycle. A failing test can cover up other
regressions, so skip this known failing test to fix the builder.

For #35236.
For #36181.

Change-Id: I07e795e75d7e37bc96ab68607d5d5cc9254342f8
Reviewed-on: https://go-review.googlesource.com/c/go/+/223780
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
gopherbot pushed a commit that referenced this issue Mar 31, 2020
These non-short Windows test failures were resolved fully in CL 206144.

Both TestScript/build_trimpath and TestScript/version tests can be fixed
by backporting the changes to test scripts only, so that is done here.

Fixing TestScript/mod_list_dir requires backporting non-test changes in
addition to the test script changes, which is unlikely to be appropriate
this late in Go 1.13 release cycle. A failing test can cover up other
regressions, so skip this known failing test to fix the builder.

For #36181.

Change-Id: I4f140bd373554eb4664f04638666dee77986ec3e
Reviewed-on: https://go-review.googlesource.com/c/go/+/223782
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.