Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better tags for spec level includes/excludes #1820

Closed
Tracked by #2226
sksamuel opened this issue Oct 30, 2020 · 5 comments · Fixed by #2285
Closed
Tracked by #2226

Better tags for spec level includes/excludes #1820

sksamuel opened this issue Oct 30, 2020 · 5 comments · Fixed by #2285
Labels
enhancement ✨ Suggestions for adding new features or improving existing ones. pinned 📌 Issues of high importance or that need to remain visible.
Milestone

Comments

@sksamuel
Copy link
Member

sksamuel commented Oct 30, 2020

The current @tags on a spec is powerful but can be confusing. A spec annotated with @tags("Foo") will not necessarily be excluded if you run with kotest.tags=Bar because root level tests may be annotated with Bar and so it must instantiate the spec to check. Only if a tag is explicitly excluded does the @tags on a spec help.

This ticket is to:

  • Add a new annotation (name whatever, say @tagged) which, when an include is set, is required, and when an exclude is set, must not be present, for that spec to be instantiated.
  • deprecate @tags.
@sksamuel sksamuel added this to the 4.4 milestone Oct 30, 2020
@kopper
Copy link
Contributor

kopper commented Oct 30, 2020

To give you some context let me describe our use-case first. As many other projects we have unit and integration tests. We would like to be able to run only one set of them. Right now we are using naming convention *Test for uni tests and *IT for integration and maven surefire plugin to run unit tests while maven failsafe plugin to run integration tests. I expect us to define more test types in the future and classify some as "slow" to avoid running them too frequently.

In order to save time/CPU we'd like to avoid instantiating specs that are not of the type that was requested to run. I understand that in order to determine all tags associated with the test we need to instantiate the spec.

What would work in our case is to have distinction between test type and test tag, for example:

@TestType("UnitTest")
class MyTest : FunSpec({
  tags(SlowTest, FlakyTest)
}

and then -Dkotest.type=IntegrationTest wouldn't even instantiate the spec.

@sksamuel
Copy link
Member Author

sksamuel commented Oct 31, 2020

Your use case is the perfect example of what I was trying to convey.

Although rather than introduce two types of tags/things, I think just having a different annotation may be easier, because then it expands out to any use case. For example @jschneidereit uses a different tag per team.

@RequiresTag(UnitTest) // tag must be present to be instantiated
class MyTest1 : FunSpec({
  test("a").config(tag = FlakyTest) { }
  test("b").config(tag = SlowTest) { }
}

@Tags(FooTag) // a tag that is simply added to the other tags of the tests
class MyTest2 : FunSpec({
  test("c").config(tag = FlakyTest) { }
  test("d").config(tag = SlowTest) { }
}

MyTest1 will now only be instantiated when kotest.tags explicitly includes UnitTest.

So under the following behaviors.

kotest.tags=UnitTest => a,b (but MyTest2 is instantiated to search inside)
kotest.tags=UnitTest & FlakyTest => a (but MyTest2 is instantiated to search inside)
kotest.tags=FlakyTest => a, c
kotest.tags not specified = a, b, c, d
kotest.tags=!UnitTest=c, d (MyTest1 never instantiated)
kotest.tags=!FlakeyTest = b, d
kotest.tags=FooTag = c, d (MyTest1 never instantiated)

The name of this annotation could be something other than RequiresTag of course. And Tags could be deprecated and replaced with another tag that is more clear, such as InheritTags or TestTags or something.

^^ @SergKhram This is what you are wanting from #1823

@SergKhram
Copy link
Contributor

SergKhram commented Nov 7, 2020

@sksamuel
kotest.tags=UnitTest => a,b (but MyTest2 is instantiated to search inside)
kotest.tags=FlakyTest => a, c

How it can work? If you write @RequiresTag(UnitTest) // tag must be present to be instantiated
I think it should be only c in case kotest.tags=FlakyTest (because we need to instantiate the spec to know what tags inside - in cases)

@LeoColman LeoColman added the enhancement ✨ Suggestions for adding new features or improving existing ones. label Nov 14, 2020
@sksamuel sksamuel modified the milestones: 4.4, 4.5 Dec 13, 2020
@sksamuel sksamuel mentioned this issue Feb 7, 2021
26 tasks
@stale
Copy link

stale bot commented Feb 12, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Feb 12, 2021
@sksamuel sksamuel removed the wontfix label Feb 12, 2021
@stale
Copy link

stale bot commented Apr 15, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Apr 15, 2021
@sksamuel sksamuel added pinned 📌 Issues of high importance or that need to remain visible. and removed wontfix labels Apr 15, 2021
@sksamuel sksamuel modified the milestones: 4.5, 4.6 Apr 21, 2021
@sksamuel sksamuel modified the milestones: 4.6, 4.7 May 18, 2021
@sksamuel sksamuel mentioned this issue May 19, 2021
71 tasks
sksamuel added a commit that referenced this issue May 25, 2021
* Added @RequiresTag for improved spec exclude capability #1820

* fix

* fix

* fix

* fix

* fix

* fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement ✨ Suggestions for adding new features or improving existing ones. pinned 📌 Issues of high importance or that need to remain visible.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants