Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: test with -run /something prints "no tests to run" inconsistently #48851

Open
katiehockman opened this issue Oct 7, 2021 · 8 comments
Open

Comments

@katiehockman
Copy link
Member

@katiehockman katiehockman commented Oct 7, 2021

When running a test with -run /subtest_name where no subtests match, it will print okay, even though no tests were run. Same is true if you use -run Test/subtest_name.
If instead, you do -run TestNameInPackage/subtest_name (where TestNameInPackage is the name of a test in that package) it will print "[no tests to run]"

Example:

➜  go test archive/tar -run /nomatchinghere 
ok      archive/tar     0.145s
➜  go test archive/tar -run Test/nomatchinghere 
ok      archive/tar     0.164s
➜  go test archive/tar -run TestWrite/nomatchinghere
ok      archive/tar     0.138s [no tests to run]

It has a similar issue when -v is set

/cc @bcmills @jayconrod @matloob

@katiehockman katiehockman added this to the Go1.18 milestone Oct 7, 2021
@bcmills
Copy link
Member

@bcmills bcmills commented Oct 7, 2021

Subtests can be created dynamically, so the only way to conclusively discover all of them is to actually run the top-level tests that match the pattern.

So, in practice, -run /subtest_name means “run all of the top-level tests, and prune out any subtest that does not match subtest_name. It's certainly an odd interaction, but there isn't a clear way to avoid it with the way subtests are defined.

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 7, 2021

So I guess that means that the first two outputs are correct, and the third one is wrong.

-v shows that it actually is running some tests, and those tests pass:

$ go test -v archive/tar -run=TestWrite/nomatchinghere
=== RUN   TestWriter
--- PASS: TestWriter (0.00s)
=== RUN   TestWriterErrors
--- PASS: TestWriterErrors (0.00s)
testing: warning: no tests to run
PASS
ok      archive/tar     0.015s [no tests to run]

@katiehockman
Copy link
Member Author

@katiehockman katiehockman commented Oct 7, 2021

So I guess that means that the first two outputs are correct, and the third one is wrong.

Yep I agree. At the very least, -run Test/nomatchinghere and -run TestWriter/nomatchinghere should do the same thing, but don't. The latter prints "[no tests to run]" and the former doesn't. And like you said, TestWriter and TestWriterErrors are both being run, it's just that none of the subtests are running.

@FiloSottile
Copy link
Contributor

@FiloSottile FiloSottile commented Oct 7, 2021

OTOH, if I specified a subtest filter which didn't match anything, it's probably not what I intended to do and would like to know, so although the third output is technically wrong but IMHO preferable.

@katiehockman
Copy link
Member Author

@katiehockman katiehockman commented Oct 7, 2021

OTOH, if I specified a subtest filter which didn't match anything, it's probably not what I intended to do and would like to know, so although the third output is technically wrong but IMHO preferable.

I see your point about the fact that it probably wasn't intentionally. But a test is getting run, it's just that no subtests were run. Imagine a scenario where TestWriterErrors has a t.Log outside the t.Run. Then you see something like this:

~ go test archive/tar -run TestWriter/doesnotmatch -v 
=== RUN   TestWriter
--- PASS: TestWriter (0.00s)
=== RUN   TestWriterErrors
    writer_test.go:838: something in TestWriterErrors
--- PASS: TestWriterErrors (0.00s)
testing: warning: no tests to run
PASS
ok      archive/tar     0.622s [no tests to run]

Which is a bit confusing. I would think "Why is it printing the log line if no tests were run?". Especially knowing that if I replaced the t.Log with a t.Error, the test would fail, which shows that it definitely ran.
e.g.

~ go test archive/tar -run TestWriter/doesnotmatch -v
=== RUN   TestWriter
--- PASS: TestWriter (0.00s)
=== RUN   TestWriterErrors
    writer_test.go:838: error in TestWriterErrors
--- FAIL: TestWriterErrors (0.00s)
testing: warning: no tests to run
FAIL
FAIL    archive/tar     0.659s
FAIL

@katiehockman
Copy link
Member Author

@katiehockman katiehockman commented Oct 7, 2021

That one is even worse, because it both printed FAIL and "testing: warning no tests to run". I don't know of any other situation where that can happen.

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 7, 2021

My inclination is to change the message to [no matching subtests] if we ran at least one top-level test and some component of the -run pattern ended up not matching anything.

That would at least be correct in all three cases, even if it conveys slightly less information about whether the tests that did run were meaningful.

@katiehockman
Copy link
Member Author

@katiehockman katiehockman commented Oct 7, 2021

My inclination is to change the message to [no matching subtests] if we ran at least one top-level test and some component of the -run pattern ended up not matching anything.

Seems reasonable enough to me 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants