Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: expose testing.T constructor so a test with subtests can be verified to fail #39903

Open
eli-darkly opened this issue Jun 28, 2020 · 1 comment
Labels
Projects
Milestone

Comments

@eli-darkly
Copy link

@eli-darkly eli-darkly commented Jun 28, 2020

This is the same proposal as #28021 - but I'd like to raise it again because the proposed solution (passing in a testing.TB instead of a *testing.T) isn't applicable to my use case.

1. Basic scenario (same as previous proposal)

I've defined a component interface which may have many implementations. All of the implementations are expected to adhere to the same contract for various standard operations. So I've created a base test suite that, given an implementation instance, runs a standard set of contract tests against it. Something like this:

func RunStandardTests(t *testing.T, implementationInstance MyInterface)

Now, if I'm going to tell everyone to use this test suite to validate their implementations, I want to make sure it is actually testing what it's supposed to be testing. So I have a test suite for the test suite (TSftTS for short). It creates a mock implementation of MyInterface which is guaranteed to adhere to the contract, and it runs RunStandardTests on that, which should pass.

However, I also need to make sure the test suite fails when it's supposed to fail, by instrumenting my mock implementation of MyInterface to break the contract in various ways. But I can't run this logic against the *testing.T that is passed into the TSftTS, because even though I would be able to detect the failure with t.Failed(), it would still cause the TSftTS itself to fail which is the opposite of what I want.

2. The previously proposed solution

Change RunStandardTests to take a testing.TB instead of a *testing.T. Create a mock implementation of testing.B which records failures, run the test suite against the deliberately-bad component with this mock implementation, and verify its state afterward.

3. My new concern

There are many tests in this test suite. So, I've used Run to produce nicely organized results:

func RunStandardTests(t *testing.T, implementationInstance MyInterface) {
    t.Run("test operation A", func(t *testing.T) { ... })
    t.Run("a bunch of tests for operation B", func (t *testing.T) {
        t.Run("B scenario 1", func (t *testing.T) { ... }) // etc., etc.
    })
}

Unfortunately, for obvious reasons the testing.TB interface does not have a Run(string, testing.TB) method.

Possible solutions

Preferred

What would be nicest from my point of view is to— as the original issue reporter requested— give me a way to create a testing.T that is not coupled to the current test environment, so if it fails, it sets its own Failed() state to true but does not cause the parent test to fail.

Workaround

One workaround would be to define yet another interface.

type MyTesting interface {
    testing.TB
    Run(string, func(MyTesting))
}

type RealTesting struct {
    t *testing.T
}

type MockTesting struct {
    failed bool
}

func (r *RealTesting) Errorf(format string, args ...interface{}) {
    r.t.Errorf(format, args...)
}
// also implement the other testing.TB methods for RealTesting here

func (r *RealTesting) Run(name string, action func(MyTesting)) {
    r.t.Run(name, func(tt *testing.T) { action(RealTesting{tt}) }
}

func (m *MockTesting) Errorf(format string, args ...interface{}) {
    m.failed = true
}

func (m *MockTesting) Run(name string, action func(MyTesting)) {
    action(m)
}

func RunStandardTests(t *testing.T, implementationInstance MyInterface) {
    runStandardTestsInternal(RealTesting{t}, implementationInstance)
}

func runStandardTestsInternal(t MyTesting implementationInstance MyInterface) {
    t.Run("test operation A", func(t MyTesting) { ... })
    t.Run("a bunch of tests for operation B", func (t MyTesting) {
        t.Run("B scenario 1", func (t MyTesting) { ... }) // etc., etc.
    })
}

func TestTheTestSuite(t *testing.T) {
    // now I want to make it fail
    badInstance := createImplementationInstanceThatBreaksTheContract
    var mockTest MockTesting
    runStandardTestsInternal(mockTest)
    assert.True(t, mockTest.failed)
}

That ought to work, but it's ungainly. It also means that if I have any test helpers that take a *testing.T, which are also used by other test code that uses *testing.T, I would have to create new versions of them that take MyTesting instead.

(I'm using go1.13.7. The rest of my environment details aren't relevant)

@gopherbot gopherbot added this to the Proposal milestone Jun 28, 2020
@gopherbot gopherbot added the Proposal label Jun 28, 2020
@eli-darkly
Copy link
Author

@eli-darkly eli-darkly commented Jun 28, 2020

Another limitation of the MyTesting workaround is that I can't use this with any test code that uses t.Parallel().

@ianlancetaylor ianlancetaylor changed the title proposal: expose testing.T constructor so a test with subtests can be verified to fail proposal: testing: expose testing.T constructor so a test with subtests can be verified to fail Jun 28, 2020
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Jun 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Incoming
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.