Repository navigation
testing: (*B).Fatal should cause non-zero exit status #14307
Description
lucky(~/src/issue) % cat issue_test.go
package issue
import "testing"
func BenchmarkFail(b *testing.B) {
for n := 0; n < b.N; n++ {
b.Fatal("non zero exit")
}
}
lucky(~/src/issue) % go test issue_test.go -test.bench=.
testing: warning: no tests to run
PASS
BenchmarkFail-4 --- FAIL: BenchmarkFail-4
issue_test.go:7: non zero exit
ok command-line-arguments 0.003s
lucky(~/src/issue) % echo $?
0
Activity
robpike commentedon Feb 12, 2016
The subject of the issue does not match the behavior of the program, and there is no explanatory text. Please explain.
davecheney commentedon Feb 12, 2016
b.Fatalshould terminate the test program like its counterpartt.Fatal[-]b.Fatal does not terminate test[/-][+]testing: b.Fatal does not terminate test[/+]robpike commentedon Feb 12, 2016
t.Fatal does not terminate the program. It terminates the test function. b.Fatal seems to do that as well, or at least your example does not demonstrate otherwise.
It looks like there is an issue that b.Fatal does not cause the program to exit with a non-zero exit code. Is that what you're saying?
davecheney commentedon Feb 12, 2016
I am sorry, you are correct. But calling t.Fatal in a test will cause the
test binary to eventually exit with a non zero error code. I was surprised
that calling the counterpart on testing.B did not cause the same behaviour.
On Sat, 13 Feb 2016, 09:32 Rob Pike notifications@github.com wrote:
cespare commentedon Feb 12, 2016
@davecheney want to change the issue title? Maybe "testing: (*B).Fatal should cause non-zero exit status".
I took a brief look at this. The code does this right now (irrelevant lines elided):
Unfortunately the exported function RunBenchmarks doesn't return failure status. Perhaps we could modify InternalBenchmark to smuggle failure status through.
If we fix this, should we also only print "PASS" if benchmarks pass, as well as tests and examples?
[-]testing: b.Fatal does not terminate test[/-][+]esting: (*B).Fatal should cause non-zero exit status[/+][-]esting: (*B).Fatal should cause non-zero exit status[/-][+]testing: (*B).Fatal should cause non-zero exit status[/+]cmarcelo commentedon Feb 26, 2016
Would adding a return value to RunBenchmarks break the compatibility promise? It seems to me it keeps the source compatibility since ignoring the single return value doesn't require any change to calling code.
bradfitz commentedon Feb 26, 2016
Yes.
bradfitz commentedon Feb 26, 2016
@cmarcelo, see http://blog.merovius.de/2015/07/29/backwards-compatibility-in-go.html
cmarcelo commentedon Feb 26, 2016
@bradfitz Thanks for the link.
I wrote a CL that adds an extra field to the struct InternalBenchmark to pass the information (a "failed bool" field). It does pass the API check in all.bash, and seems "acceptable" from reading https://golang.org/doc/go1compat document, and the Internal in the name makes me thing is not widely used. On the other hand it doesn't pass the criteria from the post you've linked. So I'm a bit unsure whether is OK or not.
Pushed the CL in https://go-review.googlesource.com/#/c/19889/.
gopherbot commentedon Feb 26, 2016
CL https://golang.org/cl/19889 mentions this issue.
cmarcelo commentedon Feb 27, 2016
Ian had a better suggestion without having to add a field to the exported struct, it became patch set 3 in the CL.
I'm still curious about the compatibility: because of the embedding explanation mentioned in the link, is adding fields a no-go for exported structs?
ianlancetaylor commentedon Feb 27, 2016
In the compatibility doc (https://golang.org/doc/go1compat) we explicitly permit adding fields to exported structs.