New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance support for dynamic tests #378

Open
sbrannen opened this Issue Jul 5, 2016 · 18 comments

Comments

@sbrannen
Member

sbrannen commented Jul 5, 2016

Status Quo

Since 5.0 M1, dynamic tests can be registered as lambda expressions, but there are a few limitations with the current feature set.

Topics

  1. Lifecycle callbacks and extensions are not applied around the invocation of a dynamic test.
  2. A dynamic test cannot directly benefit from parameter resolution.
    • A dynamic test cannot make use of a TestReporter.

Related Issues

  • #14 "Introduce support for parameterized tests"
  • #386 "Improve documentation of DynamicTest lifecycle"
  • #393 "TestReporter does not capture the correct TestIdentifier when used with DynamicTests"
  • #431 "Introduce mechanism for terminating Dynamic Tests early"
  • #694 (duplicates this issue, with example)

Deliverables

Address each topic.

@mkobit

This comment has been minimized.

Show comment
Hide comment
@mkobit

mkobit Dec 5, 2016

Contributor

I think it could also be useful/interesting for extensions in this case to do be able to compute "test matrix" by using multiple different parameter resolver extensions and different declarative parameterized values.

For example:

@Test
@ForAValue({3, 4})
@ForBValue({5, 6})
void runMyTest(AClass aValue, BClass bValue) { ... }

I would like to have the ability to have an extension that can register additional tests by looking at the context of a method or class.

Contributor

mkobit commented Dec 5, 2016

I think it could also be useful/interesting for extensions in this case to do be able to compute "test matrix" by using multiple different parameter resolver extensions and different declarative parameterized values.

For example:

@Test
@ForAValue({3, 4})
@ForBValue({5, 6})
void runMyTest(AClass aValue, BClass bValue) { ... }

I would like to have the ability to have an extension that can register additional tests by looking at the context of a method or class.

@smoyer64

This comment has been minimized.

Show comment
Hide comment
@smoyer64

smoyer64 Dec 5, 2016

Contributor

@mkobit Take a look at PR #577 and the associated proposal in issue #354 ... the idea is to be able to alter the test plan before it's made immutable.

Contributor

smoyer64 commented Dec 5, 2016

@mkobit Take a look at PR #577 and the associated proposal in issue #354 ... the idea is to be able to alter the test plan before it's made immutable.

@nicolaiparlog

This comment has been minimized.

Show comment
Hide comment
@nicolaiparlog

nicolaiparlog Dec 22, 2016

Contributor

Regarding lifecycle integration for dynamic tests... What exactly should that look like? Would the whole extension shebang (test instance processing, test parameter injection, before/after, ...) be run for test factory methods and then again for each individual dynamic test?

Mmmh, now that I'm writing this I don't actually see a problem with that anymore. But I'm sure I saw one when creating #530... Damn brain! 😫

Contributor

nicolaiparlog commented Dec 22, 2016

Regarding lifecycle integration for dynamic tests... What exactly should that look like? Would the whole extension shebang (test instance processing, test parameter injection, before/after, ...) be run for test factory methods and then again for each individual dynamic test?

Mmmh, now that I'm writing this I don't actually see a problem with that anymore. But I'm sure I saw one when creating #530... Damn brain! 😫

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Dec 22, 2016

Member

Would the whole extension shebang (test instance processing, test parameter injection, before/after, ...) be run for test factory methods and then again for each individual dynamic test?

That's certainly what I have in mind!

Member

sbrannen commented Dec 22, 2016

Would the whole extension shebang (test instance processing, test parameter injection, before/after, ...) be run for test factory methods and then again for each individual dynamic test?

That's certainly what I have in mind!

@ledoyen

This comment has been minimized.

Show comment
Hide comment
@ledoyen

ledoyen Mar 13, 2017

Contributor

Here is a proposition for lifecycle hooks invocations around dynamic tests : #735

Contributor

ledoyen commented Mar 13, 2017

Here is a proposition for lifecycle hooks invocations around dynamic tests : #735

@sormuras

This comment has been minimized.

Show comment
Hide comment
@sormuras

sormuras Jun 16, 2017

Member

Related issues updated.

Member

sormuras commented Jun 16, 2017

Related issues updated.

@paulmiddelkoop

This comment has been minimized.

Show comment
Hide comment
@paulmiddelkoop

paulmiddelkoop Feb 17, 2018

At the moment my team is using JUnit 4 in all our tests except when we we need data driven tests. We use Spock there and were very happy with table based testing using where. Writing Groovy tests inside a Java project worked well for us. But since we switched to Kotlin this does not work very well (for instance using Kotlin data classes in Groovy is kind of weak).

So we switched to JUnit 5 and see if we could use the experimental parameterized testing functionality. Personally I don't like if the parameterized data of your test is hidden away in a separate method, class or CSV file. I think one the powers of Spock is that the table is part of the test method.
Therefor I really liked the dynamic test approach where we can specify the test data on top of the test and put the real test in an lambda below.
Only thing I'm missing here is integration with lifecycles. I think it would be very nice if every single Test in a Dynamic test will have the same lifecycle as a regular @Test.
This will also make it easier to use other frameworks like Mockito in dynamic tests.

At the moment my team is using JUnit 4 in all our tests except when we we need data driven tests. We use Spock there and were very happy with table based testing using where. Writing Groovy tests inside a Java project worked well for us. But since we switched to Kotlin this does not work very well (for instance using Kotlin data classes in Groovy is kind of weak).

So we switched to JUnit 5 and see if we could use the experimental parameterized testing functionality. Personally I don't like if the parameterized data of your test is hidden away in a separate method, class or CSV file. I think one the powers of Spock is that the table is part of the test method.
Therefor I really liked the dynamic test approach where we can specify the test data on top of the test and put the real test in an lambda below.
Only thing I'm missing here is integration with lifecycles. I think it would be very nice if every single Test in a Dynamic test will have the same lifecycle as a regular @Test.
This will also make it easier to use other frameworks like Mockito in dynamic tests.

@paulmiddelkoop

This comment has been minimized.

Show comment
Hide comment
@paulmiddelkoop

paulmiddelkoop Feb 17, 2018

I don't see why @ParameterizedTest do support lifecycles and dynamic tests don't. For me it's the same concept but in a different form.

I don't see why @ParameterizedTest do support lifecycles and dynamic tests don't. For me it's the same concept but in a different form.

@sormuras

This comment has been minimized.

Show comment
Hide comment
@sormuras

sormuras Feb 17, 2018

Member

@paulmiddelkoop Please read the comments in #735 and #371 -- dynamic tests are not real tests. They are more like testlets or a visible form of the soft-assert assertAll(...) method call with its lambdas.

Perhaps you may utilize @TestTemplate or get used to @ParameterizedTest -- both take part in the full life-cycle of real tests.

Member

sormuras commented Feb 17, 2018

@paulmiddelkoop Please read the comments in #735 and #371 -- dynamic tests are not real tests. They are more like testlets or a visible form of the soft-assert assertAll(...) method call with its lambdas.

Perhaps you may utilize @TestTemplate or get used to @ParameterizedTest -- both take part in the full life-cycle of real tests.

@mmichaelis

This comment has been minimized.

Show comment
Hide comment
@mmichaelis

mmichaelis Feb 23, 2018

We just stumbled across this issue because we need the test-lifecycle for dynamic tests. We also thought about using test templates instead, but we need the test instance variables (here wired via Spring) to actually retrieve the test parameters. So our factory method looks something like this:

@ExtendWith(SpringExtension.class)
class SpringTest {
  @Autowired
  private List<Bean> beans;

  @ExtendWith(MyExtension.class)
  @TestFactory
  Stream<DynamicTest> cmTeasableTests() {
    return beans.stream().map(b -> dynamicTest("display name", () -> doTestFor(b)));
  }
}

So unless we find a solution how to access instance level fields from test-templates the test factory is much more convenient to use - but misses the important test lifecycle.

We just stumbled across this issue because we need the test-lifecycle for dynamic tests. We also thought about using test templates instead, but we need the test instance variables (here wired via Spring) to actually retrieve the test parameters. So our factory method looks something like this:

@ExtendWith(SpringExtension.class)
class SpringTest {
  @Autowired
  private List<Bean> beans;

  @ExtendWith(MyExtension.class)
  @TestFactory
  Stream<DynamicTest> cmTeasableTests() {
    return beans.stream().map(b -> dynamicTest("display name", () -> doTestFor(b)));
  }
}

So unless we find a solution how to access instance level fields from test-templates the test factory is much more convenient to use - but misses the important test lifecycle.

@sormuras

This comment has been minimized.

Show comment
Hide comment
@sormuras

sormuras Feb 23, 2018

Member

Why don't you implement doTestFor(b) in such a way that it calls custom before/after methods?

Member

sormuras commented Feb 23, 2018

Why don't you implement doTestFor(b) in such a way that it calls custom before/after methods?

@mmichaelis

This comment has been minimized.

Show comment
Hide comment
@mmichaelis

mmichaelis Feb 23, 2018

This is what I do as workaround now. But it does not fit to the extension mechanism. So what I do now is to manually create for example after each and handleTestExecutionException by using try-catch-finally. But my impression is, that this is not the expected behavior. It took my a bunch of time to realize that my tests failed, because the cleanup (in after each) was only done when all dynamic tests are done.

This is what I do as workaround now. But it does not fit to the extension mechanism. So what I do now is to manually create for example after each and handleTestExecutionException by using try-catch-finally. But my impression is, that this is not the expected behavior. It took my a bunch of time to realize that my tests failed, because the cleanup (in after each) was only done when all dynamic tests are done.

@mibutec

This comment has been minimized.

Show comment
Hide comment
@mibutec

mibutec Jun 2, 2018

For me Dynamic Tests are handled far to stepmotherly. I'd love to:

  • have Lifecycle support for them (as already discussed in this Thread)
  • have some kind of usable Surefire feedback for them (see #990)
  • discover them early (see #1338)
  • add any annotation (@Tag("myTag"), @MyVeryOwnTestAnnotation(myVeryOwnParameter=42)) to them.

mibutec commented Jun 2, 2018

For me Dynamic Tests are handled far to stepmotherly. I'd love to:

  • have Lifecycle support for them (as already discussed in this Thread)
  • have some kind of usable Surefire feedback for them (see #990)
  • discover them early (see #1338)
  • add any annotation (@Tag("myTag"), @MyVeryOwnTestAnnotation(myVeryOwnParameter=42)) to them.
@sormuras

This comment has been minimized.

Show comment
Hide comment
@sormuras

sormuras Jun 2, 2018

Member

Dynamic tests are just grouped assertions that show up in the test plan. Think of them as assertAll(...) calls on steroids! Nothing more, nothing less. I like to call them "Testlets" on my mind. Dynamic tests are good as they are. They fill a small gap that grouped assertions have: you see all assertions and you can pick a single one an re-run only that ... testlet.

Most of your requested features are already provided by normal or parameterized tests. If those two don't fit your need maybe we can extend them? Perhaps you want and need to roll your own TestTemplate implementation?

Member

sormuras commented Jun 2, 2018

Dynamic tests are just grouped assertions that show up in the test plan. Think of them as assertAll(...) calls on steroids! Nothing more, nothing less. I like to call them "Testlets" on my mind. Dynamic tests are good as they are. They fill a small gap that grouped assertions have: you see all assertions and you can pick a single one an re-run only that ... testlet.

Most of your requested features are already provided by normal or parameterized tests. If those two don't fit your need maybe we can extend them? Perhaps you want and need to roll your own TestTemplate implementation?

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Jun 3, 2018

Member

@mmichaelis,

So unless we find a solution how to access instance level fields from test-templates

What's keeping you from doing that?

Are you just saying you want to access Spring beans from within a custom test template implementation?

If so, that's already possible.

Member

sbrannen commented Jun 3, 2018

@mmichaelis,

So unless we find a solution how to access instance level fields from test-templates

What's keeping you from doing that?

Are you just saying you want to access Spring beans from within a custom test template implementation?

If so, that's already possible.

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Jun 3, 2018

Member

add any annotation (@Tag("myTag"), @MyVeryOwnTestAnnotation(myVeryOwnParameter=42)) to them.

That's unfortunately not possible in Java with Java's standard reflection APIs, and any attempt to support such annotation lookups would rely on lambda expression implementation details that may change in future JDK releases.

Member

sbrannen commented Jun 3, 2018

add any annotation (@Tag("myTag"), @MyVeryOwnTestAnnotation(myVeryOwnParameter=42)) to them.

That's unfortunately not possible in Java with Java's standard reflection APIs, and any attempt to support such annotation lookups would rely on lambda expression implementation details that may change in future JDK releases.

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Jun 3, 2018

Member

However, if you're curious about some hacky way to achieve that, feel free to take a look at my serialized lambda PoC here: sbrannen/junit-lambda-playground@ef38dde

Member

sbrannen commented Jun 3, 2018

However, if you're curious about some hacky way to achieve that, feel free to take a look at my serialized lambda PoC here: sbrannen/junit-lambda-playground@ef38dde

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Jul 30, 2018

Member

I am sad to say that the "serialized lambda" technique no longer works. See the following JDK issues for details.

It is therefore not possible to perform parameter resolution based on annotations.

Thanks to @nicolaiparlog for bringing this to my attention.

Member

sbrannen commented Jul 30, 2018

I am sad to say that the "serialized lambda" technique no longer works. See the following JDK issues for details.

It is therefore not possible to perform parameter resolution based on annotations.

Thanks to @nicolaiparlog for bringing this to my attention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment