Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: add go test -skip to skip specific tests #41583

Open
dprotaso opened this issue Sep 23, 2020 · 18 comments
Open

proposal: cmd/go: add go test -skip to skip specific tests #41583

dprotaso opened this issue Sep 23, 2020 · 18 comments

Comments

@dprotaso
Copy link

@dprotaso dprotaso commented Sep 23, 2020

t.Skip() is an option but it requires changing the test code.

It's hard to write a correct regex to pass to the -run flag in order to skip specific tests. This is due to the lack of lookaheads in the regexp library. Previous discussion here: https://groups.google.com/g/golang-nuts/c/7qgSDWPIh_E?pli=1

@dprotaso dprotaso changed the title Make it easier to skip tests with `go test` CLI Make it easier to skip tests with the `go test` CLI Sep 23, 2020
@rsc rsc added this to Incoming in Proposals Sep 23, 2020
@ianlancetaylor ianlancetaylor changed the title Make it easier to skip tests with the `go test` CLI cmd/go: make it easier to skip tests with the `go test` CLI Sep 24, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 24, 2020

Do you have a specific suggestion?

To run specific tests you can already write go test -run="TestOne|TestTwo|TestThree".

@ianlancetaylor ianlancetaylor changed the title cmd/go: make it easier to skip tests with the `go test` CLI proposal: cmd/go: make it easier to skip tests with the `go test` CLI Sep 24, 2020
@gopherbot gopherbot added this to the Proposal milestone Sep 24, 2020
@gopherbot gopherbot added the Proposal label Sep 24, 2020
@dprotaso
Copy link
Author

@dprotaso dprotaso commented Sep 24, 2020

Do you have a specific suggestion?

I didn't want to prescribe a solution but here's some off the top of my head

  1. expand run flag functionality - -run [regex] to allow look aheads - thus negation

    • pros
      • most people are familiar with this flag
    • cons
      • requires expanded regex implementation
  2. new flag -skip [regex]

    • pros
      • mirrors run flag
      • may be easier to use than a negative regex
    • cons
      • requires definition of precedence between -skip& -run
      • it'd be nice to see skipped test output - that might require more work to implement

To run specific tests you can already write go test -run="TestOne|TestTwo|TestThree".

This doesn't scale if I have O(10s) of tests in a package and want to skip one or two

@dprotaso
Copy link
Author

@dprotaso dprotaso commented Oct 6, 2020

hey @ianlancetaylor - do you have any additional questions or want me to clarify anything above?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 6, 2020

No, I'm good.

This issue is in the proposal process now and will get into the proposal committee in due course. The committee looks at a lot of issues (see minutes at #33502), so it won't be immediately, but it shouldn't take too long.

Thanks.

@rsc
Copy link
Contributor

@rsc rsc commented Oct 7, 2020

Almost no one on earth understands how to write negation regexps correctly, so I would lean toward -skip if we do this.

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 7, 2020

requires definition of precedence between -skip & -run

I can see only two reasonable definitions.

  1. Define that the default value of -run matches everything, and therefore -skip takes precedence over -run.
  2. Define that the default value of -run imposes no constraints, and therefore it is an error to specify values of -run and -skip that both match the same test.

I have a slight preference for (1), because it treats the default value of the -run flag the same as other values.

@jimmyfrasche
Copy link
Member

@jimmyfrasche jimmyfrasche commented Oct 7, 2020

Does precedence mean the order the filters are applied in or which is used if both are given?

If it's the order, then -run followed by -skip makes the most sense to me and seems more useful than disallowing the combination.

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 7, 2020

Yeah, it seems especially useful in the context of subtests:

go test foo -run=TestBar -skip=TestBar/flaky_subtest
@dprotaso
Copy link
Author

@dprotaso dprotaso commented Oct 7, 2020

Does precedence mean the order the filters are applied in or which is used if both are given?

The latter in my opinion

@rsc rsc moved this from Incoming to Active in Proposals Oct 8, 2020
@ChrisHines
Copy link
Contributor

@ChrisHines ChrisHines commented Oct 8, 2020

I have almost moved some subtests into their own top level test function on a few occasions exactly because there was not a good way to skip some subset of them. I have also had to run go test -list iterations just to figure out a regex that would run all tests except a select few, and the resulting regex is often overly long given the use case of wanting to skip a small subset when a package has a lot of tests.

@rsc
Copy link
Contributor

@rsc rsc commented Oct 14, 2020

It sounds like people are generally in favor of adding -skip with a default of "matches nothing at all".
The -skip setting applies after the -run setting (which already defaults to "match everything")
and only applies to tests (not benchmarks).

Do I have that right? Does anyone object to adding -skip as described?

@rsc rsc changed the title proposal: cmd/go: make it easier to skip tests with the `go test` CLI proposal: cmd/go: add go test -skip to skip specific tests Oct 14, 2020
@bcmills
Copy link
Member

@bcmills bcmills commented Oct 14, 2020

The -skip setting … only applies to tests (not benchmarks).

Hmm. It would be unfortunate to have an easy way to skip specific tests but not specific benchmarks.

I think it would make sense to either have -skip apply to both tests and benchmarks, or to add a separate -benchskip or similar flag for those. (It seems a bit simpler to me to make -skip also apply to benchmarks, but I don't have a strong preference either way.)

@dprotaso
Copy link
Author

@dprotaso dprotaso commented Oct 14, 2020

Do I have that right? Does anyone object to adding -skip as described?

Sounds good to me

Hmm. It would be unfortunate to have an easy way to skip specific tests but not specific benchmarks.

Expanding the scope of -skip to cover benchmarks makes sense to me. I don't mind if this comes later if it's not a trivial thing to do.

@ChrisHines
Copy link
Contributor

@ChrisHines ChrisHines commented Oct 14, 2020

A way to skip benchmarks would be welcome as well.

The -run and -bench flags have different defaults. In my experience go test -run=^$ -bench=. is pretty common today, but would people start trying to use go test -skip=. -bench=. to mean the same thing? It seems to me a separate -benchskip flag would avoid cross pollinating the two sets and reduce the chances for surprise.

@magical
Copy link
Contributor

@magical magical commented Oct 18, 2020

Instead of introducing a new flag, could we add some syntax to the existing flag? Let a pattern starting with ! invert the sense of the match. So go test -run=!Foo|Bar would run all tests that don't match Foo or Bar, same as the proposed -skip flag.

This avoids introducing new flags (-skip and -benchskip) and avoids issues about whether -run or -skip takes precedence. -run already has special handling for / so it is already not a straight regexp.

@dprotaso
Copy link
Author

@dprotaso dprotaso commented Oct 19, 2020

-run already has special handling for / so it is already not a straight regexp.

Truthfully I was tripped up by this when I first encountered it

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 19, 2020

@magical, further complications in the syntax for the -skip flag would be pretty awkward for the “run a specific test but skip a specific subtest” use-case mentioned previously.

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 19, 2020

@ChrisHines, I think the question is, would someone running go test -bench=BenchmarkFoo -skip=BenchmarkFoo/some_subbench expect it to skip BenchmarkFoo/some_subbench?

I would argue that pretty much everyone would expect that to skip the sub-benchmark, so I suspect that a unified -skip flag would be less confusing than a separate -benchskip flag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.