Description
I want to write many tests. Because I run CI, I want them all to pass all the time. Because I have not implemented all of my features, some of them actually fail: I want to skip them. Because over time I want to fix my features, I want to be notified when formerly failing tests begin passing so I can stop skipping them.
There is no good way to do this with the go testing package. I think this is a much better thing to do than just skip as long as the failures are deterministic.
My first thought is to do something like in #16502:
// ShouldFail is meant to be used as: `defer ShouldFail(t)`; it allows the
// test to still be run even when we expect it to fail.
func ShouldFail(t *testing.T) {
if t.Failed() {
t.Skip("this test is known to currently be broken")
} else {
t.Error("expected a failure, but test exited successfully")
}
}This doesn't work, because Skip doesn't un-fail a test, which is reasonable.
My second thought was to write a wrapper for *testing.T which overrides implementation for Error, Fatal, etc methods. This is not good because it's viral: I can't pass the type into functions that were expecting *testing.T as a parameter, because testing.T is type struct.
I can think of a number of ways to solve the problem if I can change the testing.T type:
- Add a "CancelFailure()" method that just clears the failure bit.
- Add a top-level "ShouldFail()" method instead of making people write it.
I can think of more invasive changes, too, but they're not worth mentioning due to the cost of change.