proposal: testing: verify failure (rather than skip) #28488

I want to write many tests. Because I run CI, I want them all to pass all the time. Because I have not implemented all of my features, some of them actually fail: I want to skip them. Because over time I want to fix my features, I want to be notified when formerly failing tests begin passing so I can stop skipping them.

There is no good way to do this with the go testing package. I think this is a much better thing to do than just skip as long as the failures are deterministic.

My first thought is to do something like in #16502:

// ShouldFail is meant to be used as: `defer ShouldFail(t)`; it allows the
// test to still be run even when we expect it to fail.
func ShouldFail(t *testing.T) {
        if t.Failed() {
                t.Skip("this test is known to currently be broken")
        } else {
                t.Error("expected a failure, but test exited successfully")
        }
}

This doesn't work, because Skip doesn't un-fail a test, which is reasonable.

My second thought was to write a wrapper for *testing.T which overrides implementation for Error, Fatal, etc methods. This is not good because it's viral: I can't pass the type into functions that were expecting *testing.T as a parameter, because testing.T is type struct.

I can think of a number of ways to solve the problem if I can change the testing.T type:

• Add a "CancelFailure()" method that just clears the failure bit.
• Add a top-level "ShouldFail()" method instead of making people write it.

I can think of more invasive changes, too, but they're not worth mentioning due to the cost of change.