Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: support running subtests of different root tests using 'go test -run' #39904

Open
dnephin opened this issue Jun 28, 2020 · 2 comments
Labels
Projects
Milestone

Comments

@dnephin
Copy link
Contributor

@dnephin dnephin commented Jun 28, 2020

The go test -run flag can be used to select multiple root test cases. This example will run the 3 tests which match the regex exactly:

go test -run '^(TestOne|TestTwo|TestThree)$'

As far as I can tell, the logic for handling subtests makes it impossible to do the same with subtest cases. It seems to be possible to run multiple root test cases, with a single subtest case:

go test -run '^TestOne$|^TestTwo$|^TestThree$/^SubTestA$'

But adding anything after the / is assumed to be a sub-sub test, and does not match the same way.

Some other things that I tried, which do not work:

Matches any SubTestA, not just the one from TestThree

^(TestOne|TestTwo|TestThree)$/^SubTestA$

Does not match any subtests

Because of how parenthesis are handled in splitRegex

(^TestOne$|^TestTwo$|^TestThree$/^SubTestA$)

Would it be reasonable to support this with the -run flag? Looking over the matching logic in src/testing it seems like this may not be a small change.

@gopherbot gopherbot added this to the Proposal milestone Jun 28, 2020
@gopherbot gopherbot added the Proposal label Jun 28, 2020
dnephin added a commit to dnephin/gotestsum that referenced this issue Jun 28, 2020
If a root test case has failed subtests, only rerun those that have
failed. A new hidden flag is added to restore the old behaviour of
re-running the entire root test case if any subtest fails.

Because of limitations of the 'go test -run' flag (golang/go#39904)
this required running each test in a separate 'go test' process.
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 28, 2020

CC @mpvl

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Jun 28, 2020
dnephin added a commit to dnephin/gotestsum that referenced this issue Jun 29, 2020
If a root test case has failed subtests, only rerun those that have
failed. A new hidden flag is added to restore the old behaviour of
re-running the entire root test case if any subtest fails.

Because of limitations of the 'go test -run' flag (github.com/golang/go/issues/39904)
this required running each test in a separate 'go test' process.
@dnephin
Copy link
Contributor Author

@dnephin dnephin commented Jul 4, 2020

I see there was some discussion about this in https://go-review.googlesource.com/c/go/+/19122/. I think the | character is not accounted for (none of the tests include a case with one). I can't see any way to make this work splitting the regex by /.

Maybe the same goal can be accomplished by adding [^/]* before and after any non-anchored / in the regex? I guess that only solves part of it. I'm not sure how to have parent tests match the regex so that all subtests are evaluated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Incoming
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.