Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: provide a way to more easily test test utility #42827

Closed
jhenstridge opened this issue Nov 25, 2020 · 9 comments
Closed

proposal: testing: provide a way to more easily test test utility #42827

jhenstridge opened this issue Nov 25, 2020 · 9 comments
Labels
Projects
Milestone

Comments

@jhenstridge
Copy link

@jhenstridge jhenstridge commented Nov 25, 2020

For larger Go projects, it's common to write some utility functions or types for use in it's test suite. Ideally, those utilities should themselves be tested. However, there are certain types of behaviour that are difficult to test at present:

  1. Check that a function causes a test to fail.
  2. Check that a function logs a particular message.
  3. Check that a function spawns an expected set of subtests.

With most testing frameworks, the usual method to handle this would be to invoke the test runner with a custom set of tests, and then inspect the results. That's not really possible with Go's testing framework, since the method of constructing a test suite and invoking the runner is all private API (either inaccessible like testing/internal/testdeps, or documented to be unstable like testing.TestMain). Also, the runner depends on global state through flags that would be shared with the real test suite.

@gopherbot gopherbot added this to the Proposal milestone Nov 25, 2020
@gopherbot gopherbot added the Proposal label Nov 25, 2020
@mvdan
Copy link
Member

@mvdan mvdan commented Nov 25, 2020

A proposal is generally a very specific idea. This seems more like a vague problem statement, and I'm not understanding what solution you're proposing.

@jhenstridge
Copy link
Author

@jhenstridge jhenstridge commented Nov 26, 2020

I guess I tagged it wrong. I was trying to describe what features I wanted without locking down an implementation.

I guess a more concrete proposal would be something like:

  1. Add a publicly supported version of testing.MainStart that doesn't depend on the undocumented testDeps interface, and takes some kind of config argument to use in lieu of the command line flags (i.e. we don't want the -run command line argument to restrict what tests are run in this independent test runner).

  2. Add a method to testing.M to set where to send test output instead of stdout. This could be a simple io.Writer accepting the default test output format, or more usefully something accepting the -json style TestEvent structures.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Nov 26, 2020

@jhenstridge would a better title for this issue be “provide a way to test Testmain functions” ?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 26, 2020

Perhaps related to #40984.

@ianlancetaylor ianlancetaylor changed the title Proposal: provide a way to more easily test test utility proposal: testing: provide a way to more easily test test utility Nov 26, 2020
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Nov 26, 2020
@jhenstridge
Copy link
Author

@jhenstridge jhenstridge commented Nov 26, 2020

@davecheney: the scope is a bit more than what a TestMain function currently does. It's the stuff that happens before TestMain is called, and the result reporting within M.Run() that I'd like access to.

The concrete use case I have for this is go-check/check#122. I was trying to port go-check to use the stdlib test runner. While I was easily able to port the code over to using stdlib subtests, porting the tests has proved more difficult. These tests took the form of tests that create a new test runner that runs a collection of tests not part of the main test suite, and then inspecting the results. That kind of thing is all private in the stdlib testing framework.

@ianlancetaylor: the suggestion in that bug report of using cleanup functions gets close to the first use case from my original post, with something like:

func TestUtility(t *testing.T) {
    t.Cleanup(func() {
        if (!t.Faled()) {
            t.Error("utility function did not fail the test")
            return
        }
        // XXX: mark test as successful somehow?
    })
    failTheTest(t)
}

This would be enough to make the test fail if failTheTest doesn't set the failed state, but it also reports a failure if things work correctly. Perhaps some function doing the opposite of T.Fail could help in this specific case.

I'm not sure it would help out with the other two use cases I mentioned, where it is not clear how you'd check they worked directly from a test calling them.

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 30, 2020

@jhenstridge, note that a test can invoke its own executable as a subprocess using the os/exec package. In many cases that suffices for testing test helpers.

(See also #41980 (comment), #39903 (comment).)

@jhenstridge
Copy link
Author

@jhenstridge jhenstridge commented Nov 30, 2020

@bcmills: Thanks for the suggestion. I'll see if I can use that to adapt the tests I'm working on. Presumably something like this:

  1. Define a test that contains the logic under test that calls t.SkipNow() unless a particular environment variable is set.
  2. Run os.Args[0] as a subprocess with the environment variable set and -run to select just the test in question. Maybe also -json if the I need the structured output.

Hopefully the tests I actually want to keep can fit into environment variables + log inspection as the only communication needed.

It would still be nice to have a more direct method of testing functionality like this.

@rsc
Copy link
Contributor

@rsc rsc commented Dec 2, 2020

This is not a concrete proposal, and we just spent a lot of time discussing the topic on #40984. In the absence of a concrete proposal and to avoid duplicated effort, closing.

@rsc rsc closed this Dec 2, 2020
@rsc rsc moved this from Incoming to Declined in Proposals Dec 2, 2020
@dnephin
Copy link
Contributor

@dnephin dnephin commented Dec 2, 2020

@jhenstridge I've had success testing test helpers in https://github.com/gotestyourself/gotest.tools so I thought I would share in case it helps you.

For the first two cases in the description I think the common practice is to have test helpers use a smaller interface instead of using testing.T. Once they accept their own interface in place of testing.T a fakeT can pretty easily confirm that Fatal, Error, or Log are called correctly.

"Check that a function spawns an expected set of subtests" is still difficult because T.Run always passes a *testing.T to the subtest. I believe that problem was discussed in #39903.

As long as go-check doesn't expose t.Run directly, it seems like it would be possible to replace it with a testing shim and call the subtest with a fakeT that records the hierarchy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Declined
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.