Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Listener to run before/after a set of tests #458

Closed
alisabzevari opened this issue Nov 1, 2018 · 16 comments
Closed

Listener to run before/after a set of tests #458

alisabzevari opened this issue Nov 1, 2018 · 16 comments
Assignees
Milestone

Comments

@alisabzevari
Copy link

@alisabzevari alisabzevari commented Nov 1, 2018

I want to have a different set of tests (probably specified by different tags) and have hooks to run before and after running those set of tests. Is it possible to do this in kotlintest?

One use case for this feature is separating integration tests from unit tests. Usually, integration tests need to set up a testing environment which takes time and it is not desired to run this set up process before all the tests (beforeProject listener).

@sksamuel

This comment has been minimized.

Copy link
Member

@sksamuel sksamuel commented Nov 7, 2018

Can you put your integration tests into separate files, and have a "before file" "after file" setup methods? We call those beforeSpec and afterSpec.

@alisabzevari

This comment has been minimized.

Copy link
Author

@alisabzevari alisabzevari commented Nov 7, 2018

I want to have a piece of code running before and after a set of tests.

One realworld example would be preparing the database schema (run migration scripts) for testing. After running migration script I want to only clean the tables (but not the whole database). Another one would be starting the mocked dependent servers in integration tests.

@alisabzevari

This comment has been minimized.

Copy link
Author

@alisabzevari alisabzevari commented Nov 7, 2018

In JUnit, it is possible by implementing an extension and extend a test class with that extension. An example is available here.

@sksamuel

This comment has been minimized.

Copy link
Member

@sksamuel sksamuel commented Nov 7, 2018

Yep, I'm just asking how you intend to make a "set". Because one way is to put each set into it's own file, in which case beforeSpec and afterSpec can work, and those can be implemented in an extension like class called a TestListener. If that doesn't work we can think of something else?

@alisabzevari

This comment has been minimized.

Copy link
Author

@alisabzevari alisabzevari commented Nov 7, 2018

If there was a way to not limit to one file, that would be great. I should still be able to freely organize my tests.

@sksamuel sksamuel mentioned this issue Nov 26, 2018
17 of 17 tasks complete
@sksamuel

This comment has been minimized.

Copy link
Member

@sksamuel sksamuel commented Nov 28, 2018

You can make a test listener class, and then apply it to every test file that you need it in. Does work?

@alisabzevari

This comment has been minimized.

Copy link
Author

@alisabzevari alisabzevari commented Nov 30, 2018

I was expecting TestListener to have a hook for this. Here is what I expect:

object Listener1: TestListener {
  override fun beforeTestSet() {
    print("test set start...")
  }
}

class Spec1: StringSpec() {
  init {
    "test 1"() { ... }
    "test 2"() { ... }
  }
}

class Spec2: StringSpec() {
  override fun listeners() = listOf(Listener1)
  init {
    "test 1"() { ... }
    "test 2"() { ... }
  }
}

class Spec3: StringSpec() {
  override fun listeners() = listOf(Listener1)
  init {
    "test 1"() { ... }
    "test 2"() { ... }
  }
}

What I expect is:

run Spec1."test 1"
run Spec1."test 2"
test set start...
run Spec2."test 1"
run Spec2."test 2"
run Spec3."test 1"
run Spec3."test 2"

As you can see, that hook will be called only one time for all of the tests having that listener. The use case is to start the integration tests facilities (like the mock servers, database etc) once for all of the integration tests.
Not to mention that, I have implemented an almost hacky way of what I wanted in this repo. I implemented an object with a lazy field and the first time I get that lazy field, the whole integration facilities will be initialized. and in the afterProject hook I clean it up.

@sksamuel

This comment has been minimized.

Copy link
Member

@sksamuel sksamuel commented Dec 2, 2018

Ok I see what you want now. The listener should be "shared" between all the classes that use it.
I'll have a think about how we can support this for 3.2

@sksamuel sksamuel added this to the 3.2 milestone Dec 2, 2018
@sksamuel sksamuel added the enhancement label Dec 2, 2018
@sksamuel

This comment has been minimized.

Copy link
Member

@sksamuel sksamuel commented Dec 3, 2018

You could do this by having a var inside the listener,

var initialized:Boolean = false

And each time beforeSpec is called, only initialise if the field is false.
And then to clean up you could use afterProject`.

Would that work ?

@sksamuel

This comment has been minimized.

Copy link
Member

@sksamuel sksamuel commented Dec 3, 2018

Another idea, is we can introduce "groups" and allow each spec to have a single group defined (or null the default). Then we can have two callbacks:

beforeGroup(group:GroupName)
afterGroup(group:GroupName)

Which would be executed only once for any given group name.

@sksamuel

This comment has been minimized.

Copy link
Member

@sksamuel sksamuel commented Dec 3, 2018

@alisabzevari

This comment has been minimized.

Copy link
Author

@alisabzevari alisabzevari commented Dec 5, 2018

You could do this by having a var inside the listener, ...

This is what I've done in the sample project that I linked in one of the comments.

@sksamuel sksamuel modified the milestones: 3.2, 3.3 Jan 22, 2019
@sksamuel sksamuel modified the milestones: 3.3, 3.4 Feb 13, 2019
@sksamuel sksamuel modified the milestones: 3.4, 3.5 Jul 13, 2019
@Raibaz

This comment has been minimized.

Copy link

@Raibaz Raibaz commented Oct 25, 2019

Another possible way to do this would be to have something like tags and categories as in JUnit.

@Kerooker

This comment has been minimized.

Copy link
Member

@Kerooker Kerooker commented Oct 25, 2019

We do have tags and theoretically that enables categories to place tests in. However, I don't think that's a solution to the issue of grouping tests and executing something before and after without boilerplate.

We don't want an user to create an extension to solve this issue, for example.

We want to create these groups

@sksamuel

This comment has been minimized.

Copy link
Member

@sksamuel sksamuel commented Jan 12, 2020

In 4.0 you can add listeners to test factories (composable dynamic tests) and these listeners only act on the tests in that composable unit.

@sksamuel sksamuel modified the milestones: 3.5, 4.0 Jan 12, 2020
@sksamuel sksamuel self-assigned this Jan 12, 2020
@sksamuel sksamuel mentioned this issue Jan 12, 2020
38 of 64 tasks complete
@sksamuel sksamuel added the framework label Jan 12, 2020
sksamuel added a commit that referenced this issue Jan 12, 2020
@sksamuel

This comment has been minimized.

Copy link
Member

@sksamuel sksamuel commented Jan 12, 2020

This now works in master and is part of 4.0
Any beforeTest / afterTest methods added to a test factory only apply to the tests in that factory.

@sksamuel sksamuel closed this Jan 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.