New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce first-class support for scenario tests #48

Open
sbrannen opened this Issue Dec 3, 2015 · 61 comments

Comments

@sbrannen
Member

sbrannen commented Dec 3, 2015

Proposal

The current proposal is to introduce the following:

  • @ScenarioTest: class-level annotation used to denote that a test class contains steps that make up a single scenario test.
  • @Step: method-level annotation used to denote that a test method is a single step within the scenario test.

The @Step annotation will need to provide an attribute that can be used to declare the next step within the scenario test. Steps will then be ordered according to the resulting dependency graph and executed in exactly that order.

For example, a scenario test could look similar to the following:

@ScenarioTest
class WebSecurityScenarioTest {

    @Step(next = "login")
    void visitPageRequiringAuthorizationWhileNotLoggedIn() {
        // attempt to visit page which requires that a user is logged in
        // assert user is redirected to login page
    }

    @Step(next = "visitSecondPageRequiringAuthorizationWhileLoggedIn")
    void login() {
        // submit login form with valid credentials
        // assert user is redirected back to previous page requiring authorization
    }

    @Step(next = "logout")
    void visitSecondPageRequiringAuthorizationWhileLoggedIn() {
        // visit another page which requires that a user is logged in
        // assert user can access page
    }

    @Step(next = END)
    void logout() {
        // visit logout URL
        // assert user has been logged out
    }

}

Related Issues

@sbrannen sbrannen self-assigned this Dec 3, 2015

@sbrannen sbrannen added this to the Alpha M1 milestone Dec 3, 2015

@mmerdes

This comment has been minimized.

Show comment
Hide comment
@mmerdes

mmerdes Dec 4, 2015

Contributor

@sbrannen: do you prefer referencing the next step by name? i assume these references would be refactoring-safe, so no real problem here. but what about typos? (i guess just using numbered steps will be awkward if you introduce steps in the middle...)

Contributor

mmerdes commented Dec 4, 2015

@sbrannen: do you prefer referencing the next step by name? i assume these references would be refactoring-safe, so no real problem here. but what about typos? (i guess just using numbered steps will be awkward if you introduce steps in the middle...)

@jlink

This comment has been minimized.

Show comment
Hide comment
@jlink

jlink Dec 4, 2015

Contributor

Is it in scope for Alpha1 (end of next week)?

Contributor

jlink commented Dec 4, 2015

Is it in scope for Alpha1 (end of next week)?

@bechte

This comment has been minimized.

Show comment
Hide comment
@bechte

bechte Dec 4, 2015

Contributor

Thanks for writing down the concept and providing a detailed example. I think it is great and I also like the next attribute pointing to the methodName. Two reasons:

  • IDE refactoring of the method name should also replace the next attribute string
  • IDE vendors will have a nice and easy way to support auto-completion for steps

I still wonder if we want to point to the next or rather to the previous step? From a reader's perspective, I would like to have an idea what are the preconditions for this step? Especially as the annotation stands above the method declaration, I would suggest using a "previous" or "ancester" attribute. But it's just a feeling.

Concerning the Alpha-M1: I don't think it will be part of it. We should probably move it to a different milestone. What do you think, @sbrannen ?

Contributor

bechte commented Dec 4, 2015

Thanks for writing down the concept and providing a detailed example. I think it is great and I also like the next attribute pointing to the methodName. Two reasons:

  • IDE refactoring of the method name should also replace the next attribute string
  • IDE vendors will have a nice and easy way to support auto-completion for steps

I still wonder if we want to point to the next or rather to the previous step? From a reader's perspective, I would like to have an idea what are the preconditions for this step? Especially as the annotation stands above the method declaration, I would suggest using a "previous" or "ancester" attribute. But it's just a feeling.

Concerning the Alpha-M1: I don't think it will be part of it. We should probably move it to a different milestone. What do you think, @sbrannen ?

@jlink

This comment has been minimized.

Show comment
Hide comment
@jlink

jlink Dec 4, 2015

Contributor

Why not use preconditions as the concept to determine the order? A step can
have none, one or many. A runner could thereby parallelize test execution
as much as possible. And one could determine the minimum set of steps that
must be executed for a certain step to make sense at all.

2015-12-04 11:24 GMT+01:00 Stefan Bechtold notifications@github.com:

Thanks for writing down the concept and providing a detailed example. I
think it is great and I also like the next attribute pointing to the
methodName. Two reasons:

  • IDE refactoring of the method name should also replace the next
    attribute string
  • IDE vendors will have a nice and easy way to support auto-completion
    for steps

I still wonder if we want to point to the next or rather to the previous
step? From a reader's perspective, I would like to have an idea what are
the preconditions for this step? Especially as the annotation stands above
the method declaration, I would suggest using a "previous" or "ancester"
attribute. But it's just a feeling.


Reply to this email directly or view it on GitHub
#48 (comment)
.

Contributor

jlink commented Dec 4, 2015

Why not use preconditions as the concept to determine the order? A step can
have none, one or many. A runner could thereby parallelize test execution
as much as possible. And one could determine the minimum set of steps that
must be executed for a certain step to make sense at all.

2015-12-04 11:24 GMT+01:00 Stefan Bechtold notifications@github.com:

Thanks for writing down the concept and providing a detailed example. I
think it is great and I also like the next attribute pointing to the
methodName. Two reasons:

  • IDE refactoring of the method name should also replace the next
    attribute string
  • IDE vendors will have a nice and easy way to support auto-completion
    for steps

I still wonder if we want to point to the next or rather to the previous
step? From a reader's perspective, I would like to have an idea what are
the preconditions for this step? Especially as the annotation stands above
the method declaration, I would suggest using a "previous" or "ancester"
attribute. But it's just a feeling.


Reply to this email directly or view it on GitHub
#48 (comment)
.

@bechte

This comment has been minimized.

Show comment
Hide comment
@bechte

bechte Dec 4, 2015

Contributor

Yes, this will be a consequence of turning from "next" to "ancestor". Preconditions are great, but we need to be aware of the added complexity. Maybe we discuss this topic in person? Anyways, I like this feature. 👍

Contributor

bechte commented Dec 4, 2015

Yes, this will be a consequence of turning from "next" to "ancestor". Preconditions are great, but we need to be aware of the added complexity. Maybe we discuss this topic in person? Anyways, I like this feature. 👍

@dinesh707

This comment has been minimized.

Show comment
Hide comment
@dinesh707

dinesh707 Dec 4, 2015

Referring a method by string sounds like way to do more mistakes and as @bechte commented it will not give IDE support when coding. I would suggest something like following

final String LOGIN_ID = "login_id";
final String LOGOUT_ID = "logout_id";
final String VISIT_PAGE_ID = "visit_page_id";

@step(id = LOGOUT_ID, waitFor = VISIT_PAGE)

so the waitFor will wait till VISIT_PAGE is completed. That way executor should be able to plan concurrent executions as well.

But again I really see the benefit of the original proposal

dinesh707 commented Dec 4, 2015

Referring a method by string sounds like way to do more mistakes and as @bechte commented it will not give IDE support when coding. I would suggest something like following

final String LOGIN_ID = "login_id";
final String LOGOUT_ID = "logout_id";
final String VISIT_PAGE_ID = "visit_page_id";

@step(id = LOGOUT_ID, waitFor = VISIT_PAGE)

so the waitFor will wait till VISIT_PAGE is completed. That way executor should be able to plan concurrent executions as well.

But again I really see the benefit of the original proposal

@sormuras

This comment has been minimized.

Show comment
Hide comment
@sormuras

sormuras Dec 4, 2015

Member

Utilizing a simple state machine (using enums instead of strings) is an overkill, right?
Like https://github.com/oxo42/stateless4j or others.

Member

sormuras commented Dec 4, 2015

Utilizing a simple state machine (using enums instead of strings) is an overkill, right?
Like https://github.com/oxo42/stateless4j or others.

@bechte

This comment has been minimized.

Show comment
Hide comment
@bechte

bechte Dec 4, 2015

Contributor

I was trying to say that I like the method name references. I would like to avoid introducing another ID for the tests. I think IDE vendors are great for picking up method names (outline view, etc.) and they will easily support refactorings and support test writers with auto-completion. This can all happen on static code analysis and, therefore, is something they can support right-away.

At least as a test writer, I dont want to maintain the list of IDs. It's boilerplate anyways. We might think about support the test name:

@Test(name = "Step1") 

and allow referencing the name as well. But for a first alpha, I would stick with the easy method name approach provided by @sbrannen .

Contributor

bechte commented Dec 4, 2015

I was trying to say that I like the method name references. I would like to avoid introducing another ID for the tests. I think IDE vendors are great for picking up method names (outline view, etc.) and they will easily support refactorings and support test writers with auto-completion. This can all happen on static code analysis and, therefore, is something they can support right-away.

At least as a test writer, I dont want to maintain the list of IDs. It's boilerplate anyways. We might think about support the test name:

@Test(name = "Step1") 

and allow referencing the name as well. But for a first alpha, I would stick with the easy method name approach provided by @sbrannen .

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Dec 4, 2015

Member

@jlink and @bechte, scheduling this issue for Alpha M1 was intentionally optimistic.

Whether or not it's possible for Alpha M1 really depends on how far we get with the rewrite of the JUnit 5 engine -- for example, whether we have extension points for test instance lifecycle and ordering of tests.

If we don't get far enough in Alpha M1, we will naturally push the implementation of this issue to a subsequent milestone. However, we need to make sure that everyone has this on his radar; otherwise, it will likely be more challenging to integrate this feature as an afterthought.

Member

sbrannen commented Dec 4, 2015

@jlink and @bechte, scheduling this issue for Alpha M1 was intentionally optimistic.

Whether or not it's possible for Alpha M1 really depends on how far we get with the rewrite of the JUnit 5 engine -- for example, whether we have extension points for test instance lifecycle and ordering of tests.

If we don't get far enough in Alpha M1, we will naturally push the implementation of this issue to a subsequent milestone. However, we need to make sure that everyone has this on his radar; otherwise, it will likely be more challenging to integrate this feature as an afterthought.

@dsaff

This comment has been minimized.

Show comment
Hide comment
@dsaff

dsaff Dec 4, 2015

Member

If this wasn't first-class supported, how hard would it be to add via a
current extension point?

David Saff

Member

dsaff commented Dec 4, 2015

If this wasn't first-class supported, how hard would it be to add via a
current extension point?

David Saff

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Dec 4, 2015

Member

@mmerdes, @bechte, @jlink, @dinesh707,

Thanks for the feedback on ordering and dependencies!

The reason I have thus far only proposed an @Step annotation with a next attribute is to keep it as simple as possible for the first round of implementation.

To address the issues you have raised...

Depending on how we decide to support ordering in general (i.e., for standard @Test methods), we may find a common mechanism for ordering steps in a scenario test as well. For example, if we introduce something like @Order for @Test methods, we could optionally reuse that for @Step methods.

The difference is that an annotation like @Order is typically based on numerical ordering. If you initially, intentionally define gaps between the numbers you declare (e.g., 10, 20, 30 instead of 1, 2, 3), it is then possible to easily insert another test or step in one of those gaps. In that sense numerical ordering is rather flexible though at the same time not as precise. For steps in a scenario test, however, I imagine people will wish to be very precise about the dependencies between steps.

As for whether a step declares its dependency by referencing the previous step vs. the next step, I think it will become more clear what the better solution is once we begin to experiment more with this feature.

With regard to supporting a precondition mechanism, that would certainly increase the flexibility of step execution within a scenario test, but I think we should first focus on a simpler approach before adding the level of complexity that would be required to support that correctly.

Along the same line of thinking, for the initial draft of this feature I would recommend that we not concern ourselves with parallel execution of steps within a scenario test.

Member

sbrannen commented Dec 4, 2015

@mmerdes, @bechte, @jlink, @dinesh707,

Thanks for the feedback on ordering and dependencies!

The reason I have thus far only proposed an @Step annotation with a next attribute is to keep it as simple as possible for the first round of implementation.

To address the issues you have raised...

Depending on how we decide to support ordering in general (i.e., for standard @Test methods), we may find a common mechanism for ordering steps in a scenario test as well. For example, if we introduce something like @Order for @Test methods, we could optionally reuse that for @Step methods.

The difference is that an annotation like @Order is typically based on numerical ordering. If you initially, intentionally define gaps between the numbers you declare (e.g., 10, 20, 30 instead of 1, 2, 3), it is then possible to easily insert another test or step in one of those gaps. In that sense numerical ordering is rather flexible though at the same time not as precise. For steps in a scenario test, however, I imagine people will wish to be very precise about the dependencies between steps.

As for whether a step declares its dependency by referencing the previous step vs. the next step, I think it will become more clear what the better solution is once we begin to experiment more with this feature.

With regard to supporting a precondition mechanism, that would certainly increase the flexibility of step execution within a scenario test, but I think we should first focus on a simpler approach before adding the level of complexity that would be required to support that correctly.

Along the same line of thinking, for the initial draft of this feature I would recommend that we not concern ourselves with parallel execution of steps within a scenario test.

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Dec 4, 2015

Member

@dsaff,

With the current set of extension points, it would be impossible to implement a feature like this as an extension.

However, that may soon change, depending on the outcome of the current effort to redesign the test discovery and test execution models.

In general, our goal is to introduce as many extension points that we deem feasible. Thus, if we create the necessary extension points in the engine to make it possible to implement scenario tests as an extension, we would naturally opt to implement it as an extension even if the feature is available out-of-the-box. That's analogous to how we implemented @Disabled and @TestName.

Member

sbrannen commented Dec 4, 2015

@dsaff,

With the current set of extension points, it would be impossible to implement a feature like this as an extension.

However, that may soon change, depending on the outcome of the current effort to redesign the test discovery and test execution models.

In general, our goal is to introduce as many extension points that we deem feasible. Thus, if we create the necessary extension points in the engine to make it possible to implement scenario tests as an extension, we would naturally opt to implement it as an extension even if the feature is available out-of-the-box. That's analogous to how we implemented @Disabled and @TestName.

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Dec 4, 2015

Member

@sormuras,

Yes, I think that using or implementing a true state machine would be overkill for this feature.

Member

sbrannen commented Dec 4, 2015

@sormuras,

Yes, I think that using or implementing a true state machine would be overkill for this feature.

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Dec 4, 2015

Member

@bechte,

I think you meant:

@Name("Step1")
@Step

instead of:

@Test(name = "Step1")

Right?

Member

sbrannen commented Dec 4, 2015

@bechte,

I think you meant:

@Name("Step1")
@Step

instead of:

@Test(name = "Step1")

Right?

@bechte

This comment has been minimized.

Show comment
Hide comment
@bechte

bechte Dec 4, 2015

Contributor

@sbrannen True 👍 - Thanks ;-)

Contributor

bechte commented Dec 4, 2015

@sbrannen True 👍 - Thanks ;-)

@luontola

This comment has been minimized.

Show comment
Hide comment
@luontola

luontola Dec 4, 2015

Here is an idea for an alternative way of defining test order, using the normal rules of code execution order: http://specsy.org/documentation.html#non-isolated-execution-model

That way requires that tests can be nested and that tests are declared as method calls which take a lambda as parameter, so it can be hard to fit it together with the JUnit style of declaring tests as method declarations.

luontola commented Dec 4, 2015

Here is an idea for an alternative way of defining test order, using the normal rules of code execution order: http://specsy.org/documentation.html#non-isolated-execution-model

That way requires that tests can be nested and that tests are declared as method calls which take a lambda as parameter, so it can be hard to fit it together with the JUnit style of declaring tests as method declarations.

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Dec 4, 2015

Member

@orfjackal, thanks for sharing the link to Specsy's Non-Isolated Execution Model.

shareSideEffects() does in fact support the same underlying runtime model that we aim to achieve; however, you are also correct that it does not align with the standard Java class and method based approach used in JUnit 4 and JUnit 5.

Since the JDK does not guarantee that methods are returned in the order in which they are declared when using reflection, we have to introduce an explicit mechanism to control the order. So unless we introduce an alternative programming model based solely on lambdas (which has in fact been discussed), we unfortunately won't be able to support a programming model as simple as Specsy in this regard.

Member

sbrannen commented Dec 4, 2015

@orfjackal, thanks for sharing the link to Specsy's Non-Isolated Execution Model.

shareSideEffects() does in fact support the same underlying runtime model that we aim to achieve; however, you are also correct that it does not align with the standard Java class and method based approach used in JUnit 4 and JUnit 5.

Since the JDK does not guarantee that methods are returned in the order in which they are declared when using reflection, we have to introduce an explicit mechanism to control the order. So unless we introduce an alternative programming model based solely on lambdas (which has in fact been discussed), we unfortunately won't be able to support a programming model as simple as Specsy in this regard.

@marcphilipp marcphilipp modified the milestones: M1, Alpha 1 Dec 11, 2015

@thomasdarimont

This comment has been minimized.

Show comment
Hide comment
@thomasdarimont

thomasdarimont Dec 16, 2015

Interesting discussion - I just wanted to throw in two alternative ideas that came to my mind:

One alternative for referring to test methods via Strings or explicit lambdas could be method-references.
It's a "bit" more involved but would look less verbose than lambdas and could also be checked by the compiler.
Here is an example for this: https://gist.github.com/thomasdarimont/eb1bc1d24ccf3decbdc6
(Though I admit that this is more of a hack...)

Another idea is to create a proxied mock object of the current test class that would allow you to specify the
order of test executions via the invocation order of the mock methods. This is similar to what Mocking frameworks
like mockito support but here one would describe the actual "execution plan" for the test:

    @ScenarioSetup
    public void simpleScenario(WebSecurityScenarioTest mock){

        mock.visitPageRequiringAuthorizationWhileNotLoggedIn();
        mock.login();
        mock.visitSecondPageRequiringAuthorizationWhileLoggedIn();
        mock.logout();
    }

thomasdarimont commented Dec 16, 2015

Interesting discussion - I just wanted to throw in two alternative ideas that came to my mind:

One alternative for referring to test methods via Strings or explicit lambdas could be method-references.
It's a "bit" more involved but would look less verbose than lambdas and could also be checked by the compiler.
Here is an example for this: https://gist.github.com/thomasdarimont/eb1bc1d24ccf3decbdc6
(Though I admit that this is more of a hack...)

Another idea is to create a proxied mock object of the current test class that would allow you to specify the
order of test executions via the invocation order of the mock methods. This is similar to what Mocking frameworks
like mockito support but here one would describe the actual "execution plan" for the test:

    @ScenarioSetup
    public void simpleScenario(WebSecurityScenarioTest mock){

        mock.visitPageRequiringAuthorizationWhileNotLoggedIn();
        mock.login();
        mock.visitSecondPageRequiringAuthorizationWhileLoggedIn();
        mock.logout();
    }
@jillesvangurp

This comment has been minimized.

Show comment
Hide comment
@jillesvangurp

jillesvangurp Dec 31, 2015

Why not rename @ScenarioTest to @Test? I don't see the need for introducing a completely new annotation. Currently junit doesn't have any class level @Test, like testng does have. I've always thought this was a strange omission and junit 5 looks like it could be a nice moment to fix that. The meaning of this is simple: every public method in this class is a test (unless annotated with BeforeXXX/AfterXXX). This would eliminate a great deal of verbosity since it allows you to leave out redundant @Test declarations on each method.

I would also change the @Step annotation to @Next("testName"). The testName should match to either the methodName or the @Name if configured. For the last step, the next would not be needed.

Any test methods with @Next would be guaranteed to be executed in the chained order. Any other tests are executed before or after in whatever default order configured for junit. Together with the class level @Test this gives you the least verbose way to do scenario testing. Combined with nested tests as proposed for junit5 this could actually provide something that closely resembles rspec.

Also having chained test methods finally gives us something like @BeforeEach and @AfterEach because each chain implictly has a first and last method without requiring the use of static methods (which I've always found a very unfortunate decision in junit). But maybe, we can make this more explicit by introducing @BeforeScenario and @AfterScenario.

jillesvangurp commented Dec 31, 2015

Why not rename @ScenarioTest to @Test? I don't see the need for introducing a completely new annotation. Currently junit doesn't have any class level @Test, like testng does have. I've always thought this was a strange omission and junit 5 looks like it could be a nice moment to fix that. The meaning of this is simple: every public method in this class is a test (unless annotated with BeforeXXX/AfterXXX). This would eliminate a great deal of verbosity since it allows you to leave out redundant @Test declarations on each method.

I would also change the @Step annotation to @Next("testName"). The testName should match to either the methodName or the @Name if configured. For the last step, the next would not be needed.

Any test methods with @Next would be guaranteed to be executed in the chained order. Any other tests are executed before or after in whatever default order configured for junit. Together with the class level @Test this gives you the least verbose way to do scenario testing. Combined with nested tests as proposed for junit5 this could actually provide something that closely resembles rspec.

Also having chained test methods finally gives us something like @BeforeEach and @AfterEach because each chain implictly has a first and last method without requiring the use of static methods (which I've always found a very unfortunate decision in junit). But maybe, we can make this more explicit by introducing @BeforeScenario and @AfterScenario.

@jlink

This comment has been minimized.

Show comment
Hide comment
@jlink

jlink Dec 31, 2015

Contributor

Sounsd like a good use case for a test engine of your own?

I wouldn't like to overload the meaning of @test within the JUnit 5 engine.

Am 31.12.2015 um 11:16 schrieb Jilles van Gurp notifications@github.com:

Why not rename @ScenarioTest to @test? I don't see the need for introducing a completely new annotation. Currently junit doesn't have any class level @test, like testng does have. I've always thought this was a strange omission and junit 5 looks like it could be a nice moment to fix that. The meaning of this is simple: every public method in this class is a test (unless annotated with Before/After). This would eliminate a great deal of verbosity since it allows you to leave out redundant @test declarations on each method.

I would also change the @step annotation to @next("testName"). The testName should match to either the methodName or the @name if configured. For the last step, the next would not be needed.

Any test methods with @next would be guaranteed to be executed in the chained order. Any other tests are executed before or after in whatever default order configured for junit. Together with the class level @test this gives you the least verbose way to do scenario testing. Combined with nested tests as proposed for junit5 this could actually provide something that closely resembles rspec.

Also having chained test methods finally gives us something like @BeforeEach and @AfterEach without requiring the use of static methods like testng provides (which I've always found a very unfortunate decision in junit). But maybe, we can make this more explicit by introducing @BeforeScenario and @AfterScenario.


Reply to this email directly or view it on GitHub.

Contributor

jlink commented Dec 31, 2015

Sounsd like a good use case for a test engine of your own?

I wouldn't like to overload the meaning of @test within the JUnit 5 engine.

Am 31.12.2015 um 11:16 schrieb Jilles van Gurp notifications@github.com:

Why not rename @ScenarioTest to @test? I don't see the need for introducing a completely new annotation. Currently junit doesn't have any class level @test, like testng does have. I've always thought this was a strange omission and junit 5 looks like it could be a nice moment to fix that. The meaning of this is simple: every public method in this class is a test (unless annotated with Before/After). This would eliminate a great deal of verbosity since it allows you to leave out redundant @test declarations on each method.

I would also change the @step annotation to @next("testName"). The testName should match to either the methodName or the @name if configured. For the last step, the next would not be needed.

Any test methods with @next would be guaranteed to be executed in the chained order. Any other tests are executed before or after in whatever default order configured for junit. Together with the class level @test this gives you the least verbose way to do scenario testing. Combined with nested tests as proposed for junit5 this could actually provide something that closely resembles rspec.

Also having chained test methods finally gives us something like @BeforeEach and @AfterEach without requiring the use of static methods like testng provides (which I've always found a very unfortunate decision in junit). But maybe, we can make this more explicit by introducing @BeforeScenario and @AfterScenario.


Reply to this email directly or view it on GitHub.

@jillesvangurp

This comment has been minimized.

Show comment
Hide comment
@jillesvangurp

jillesvangurp Dec 31, 2015

In my view it wouldn't overload @Test at all but merely give it exactly the same meaning except at the class level; just like testng has provided for some time. In junit3, you could actually extend TestCase to achieve the same goal.

The second part of what I propose is a somewhat less verbose alternative to what @sbrannen proposed and would simply allow you to chain together test methods using @Next; which is pretty much the minimum needed to support scenarios. One benefit here is that nothing else changes and all the other planned changes (e.g. dependency injection via parameters) and annotations should work here for scenarios as well.

jillesvangurp commented Dec 31, 2015

In my view it wouldn't overload @Test at all but merely give it exactly the same meaning except at the class level; just like testng has provided for some time. In junit3, you could actually extend TestCase to achieve the same goal.

The second part of what I propose is a somewhat less verbose alternative to what @sbrannen proposed and would simply allow you to chain together test methods using @Next; which is pretty much the minimum needed to support scenarios. One benefit here is that nothing else changes and all the other planned changes (e.g. dependency injection via parameters) and annotations should work here for scenarios as well.

@bechte

This comment has been minimized.

Show comment
Hide comment
@bechte

bechte Dec 31, 2015

Contributor

Three things I would like to add to the discussion:

  1. Of course we could simply use @Test on the class level to minimize the number of annotations that JUnit serves. But I don't see a good reason for doing so. We are not aiming on to minimize the number of annotations, but we are maximizing the flexibility and the expressiveness of JUnit itself. In this manner, I think it is a good thing to introduce a separate annotation that clarifies that we are testing scenarios, i.e. @ScenarioTest. I really think, this is a good thing. Anyways, we still could change the name to something else (even though I like it the way it is).

  2. We could introduce another engine that supports scenarios but this also means that this engine is required to provide its own semantics (probably using its own annotations).

  3. The @Step annotation with a next attribute is an early draft. A @Next annotation would also solve this issue of finding out what the next step is in the scenario. Still, we should also think about some form of preconditions. I prefer specifying previous steps, i.e. what are the preconditions for this actual step to run? In this way, we don't get a list of steps, but a tree of steps, which I think is more flexible. But we might find it to hard to solve the problem internally and stick by the list approach. Just to mention...

I really think we should take this issue and have a group discussion within the dev team in January.

Contributor

bechte commented Dec 31, 2015

Three things I would like to add to the discussion:

  1. Of course we could simply use @Test on the class level to minimize the number of annotations that JUnit serves. But I don't see a good reason for doing so. We are not aiming on to minimize the number of annotations, but we are maximizing the flexibility and the expressiveness of JUnit itself. In this manner, I think it is a good thing to introduce a separate annotation that clarifies that we are testing scenarios, i.e. @ScenarioTest. I really think, this is a good thing. Anyways, we still could change the name to something else (even though I like it the way it is).

  2. We could introduce another engine that supports scenarios but this also means that this engine is required to provide its own semantics (probably using its own annotations).

  3. The @Step annotation with a next attribute is an early draft. A @Next annotation would also solve this issue of finding out what the next step is in the scenario. Still, we should also think about some form of preconditions. I prefer specifying previous steps, i.e. what are the preconditions for this actual step to run? In this way, we don't get a list of steps, but a tree of steps, which I think is more flexible. But we might find it to hard to solve the problem internally and stick by the list approach. Just to mention...

I really think we should take this issue and have a group discussion within the dev team in January.

@eric-thelin

This comment has been minimized.

Show comment
Hide comment
@eric-thelin

eric-thelin Apr 2, 2016

Hi there! I enjoy reading the discussion so far. One idea that came to mind was allowing to define the step ordering in a single class-level annotation like so:

@Scenario(steps=["login", "doSomething", "logout"])
class ScenarioExample {

    @Step
    void login() {...}

    @Step
    void doSomething() {...}

    @Step
    void logout() {...}
}

Besides making it easy to see the step ordering, something I think is hard with the approaches referring to the next/previous step only, it would be possible to support definition of multiple scenarios on the same class:

@Scenario(name="Something", steps=["login", "doSomething", "logout"])
@Scenario(name="Something else", steps=["login", "doSomethingElse", "logout"])
class MultipleScenariosExample {

    @Step
    void login() {...}

    @Step
    void doSomething() {...}

    @Step
    void doSomethingElse() {...}

    @Step
    void logout() {...}
}

Thoughts?

eric-thelin commented Apr 2, 2016

Hi there! I enjoy reading the discussion so far. One idea that came to mind was allowing to define the step ordering in a single class-level annotation like so:

@Scenario(steps=["login", "doSomething", "logout"])
class ScenarioExample {

    @Step
    void login() {...}

    @Step
    void doSomething() {...}

    @Step
    void logout() {...}
}

Besides making it easy to see the step ordering, something I think is hard with the approaches referring to the next/previous step only, it would be possible to support definition of multiple scenarios on the same class:

@Scenario(name="Something", steps=["login", "doSomething", "logout"])
@Scenario(name="Something else", steps=["login", "doSomethingElse", "logout"])
class MultipleScenariosExample {

    @Step
    void login() {...}

    @Step
    void doSomething() {...}

    @Step
    void doSomethingElse() {...}

    @Step
    void logout() {...}
}

Thoughts?

@mfulton26

This comment has been minimized.

Show comment
Hide comment
@mfulton26

mfulton26 May 4, 2017

Or just use dynamicContainer directly, right?

    @TestFactory
    @DisplayName("Web Security Scenario")
    Stream<DynamicNode> scenarioTests() {
        return Stream.of(
                step("Visit page requiring authorization while not logged in", () -> {
                    // attempt to visit page which requires that a user is logged in
                    // assert user is redirected to login page
                }),
                step("Log-in", () -> {
                    // submit login form with valid credentials
                    // assert user is redirected back to previous page requiring authorization
                }),
                dynamicContainer("Can access several pages while logged in", Stream.of(
                    step("Visit second page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    }),
                    step("Visit third page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    })
                )),
                step("Log-out", () -> {
                    // visit logout URL
                    // assert user has been logged out
                })
        );
    }

With that in mind it might make more sense to name a step "dynamicStep" and then whenever test developers don't care what order tests are executed in and there is no execution dependency they can use "dynamicTest" too. e.g.:

    @TestFactory
    @DisplayName("Web Security Scenario")
    Stream<DynamicNode> scenarioTests() {
        return Stream.of(
                dynamicStep("Visit page requiring authorization while not logged in", () -> {
                    // attempt to visit page which requires that a user is logged in
                    // assert user is redirected to login page
                }),
                dynamicStep("Log-in", () -> {
                    // submit login form with valid credentials
                    // assert user is redirected back to previous page requiring authorization
                }),
                dynamicContainer("Can access several pages while logged in", Stream.of(

                    // note: ordering of pages visited within this container doesn't matter

                    dynamicTest("Visit second page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    }),
                    dynamicTest("Visit third page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    })
                )),
                dynamicStep("Log-out", () -> {
                    // visit logout URL
                    // assert user has been logged out
                })
        );
    }

mfulton26 commented May 4, 2017

Or just use dynamicContainer directly, right?

    @TestFactory
    @DisplayName("Web Security Scenario")
    Stream<DynamicNode> scenarioTests() {
        return Stream.of(
                step("Visit page requiring authorization while not logged in", () -> {
                    // attempt to visit page which requires that a user is logged in
                    // assert user is redirected to login page
                }),
                step("Log-in", () -> {
                    // submit login form with valid credentials
                    // assert user is redirected back to previous page requiring authorization
                }),
                dynamicContainer("Can access several pages while logged in", Stream.of(
                    step("Visit second page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    }),
                    step("Visit third page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    })
                )),
                step("Log-out", () -> {
                    // visit logout URL
                    // assert user has been logged out
                })
        );
    }

With that in mind it might make more sense to name a step "dynamicStep" and then whenever test developers don't care what order tests are executed in and there is no execution dependency they can use "dynamicTest" too. e.g.:

    @TestFactory
    @DisplayName("Web Security Scenario")
    Stream<DynamicNode> scenarioTests() {
        return Stream.of(
                dynamicStep("Visit page requiring authorization while not logged in", () -> {
                    // attempt to visit page which requires that a user is logged in
                    // assert user is redirected to login page
                }),
                dynamicStep("Log-in", () -> {
                    // submit login form with valid credentials
                    // assert user is redirected back to previous page requiring authorization
                }),
                dynamicContainer("Can access several pages while logged in", Stream.of(

                    // note: ordering of pages visited within this container doesn't matter

                    dynamicTest("Visit second page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    }),
                    dynamicTest("Visit third page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    })
                )),
                dynamicStep("Log-out", () -> {
                    // visit logout URL
                    // assert user has been logged out
                })
        );
    }
@sormuras

This comment has been minimized.

Show comment
Hide comment
@sormuras

sormuras May 4, 2017

Member

Mh, what about an optional flag passed to @TestFactory allowing to alter the "fail fast" or "assert all" mode? Then dynamicContainer and dynamicTest can stay as they are.

@TestFactory(RunMode.ASSERT_ALL) -- like today

@TestFactory(RunMode.FAIL_FAST) -- exit when an error occurs and mark remaining nodes as "skipped"

Member

sormuras commented May 4, 2017

Mh, what about an optional flag passed to @TestFactory allowing to alter the "fail fast" or "assert all" mode? Then dynamicContainer and dynamicTest can stay as they are.

@TestFactory(RunMode.ASSERT_ALL) -- like today

@TestFactory(RunMode.FAIL_FAST) -- exit when an error occurs and mark remaining nodes as "skipped"

@mfulton26

This comment has been minimized.

Show comment
Hide comment
@mfulton26

mfulton26 May 4, 2017

I like that. Another idea I just had would be to introduce DynamicScenario in addition to a DynamicStep. A "Scenario" is a "Container" like a "Step" is a "Test" but the prior have dependencies while the later do not. e.g.:

    @TestFactory
    @DisplayName("Web Security Scenario")
    Stream<DynamicNode> scenarioTests() {
        return Stream.of(
                dynamicStep("Visit page requiring authorization while not logged in", () -> {
                    // attempt to visit page which requires that a user is logged in
                    // assert user is redirected to login page
                }),
                dynamicStep("Log-in", () -> {
                    // submit login form with valid credentials
                    // assert user is redirected back to previous page requiring authorization
                }),
                dynamicScenario("Can access several pages while logged in", Stream.of(

                    // note: ordering of pages visited within this container doesn't matter

                    dynamicTest("Visit second page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    }),
                    dynamicTest("Visit third page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    })
                )),
                dynamicStep("Log-out", () -> {
                    // visit logout URL
                    // assert user has been logged out
                })
        );
    }

mfulton26 commented May 4, 2017

I like that. Another idea I just had would be to introduce DynamicScenario in addition to a DynamicStep. A "Scenario" is a "Container" like a "Step" is a "Test" but the prior have dependencies while the later do not. e.g.:

    @TestFactory
    @DisplayName("Web Security Scenario")
    Stream<DynamicNode> scenarioTests() {
        return Stream.of(
                dynamicStep("Visit page requiring authorization while not logged in", () -> {
                    // attempt to visit page which requires that a user is logged in
                    // assert user is redirected to login page
                }),
                dynamicStep("Log-in", () -> {
                    // submit login form with valid credentials
                    // assert user is redirected back to previous page requiring authorization
                }),
                dynamicScenario("Can access several pages while logged in", Stream.of(

                    // note: ordering of pages visited within this container doesn't matter

                    dynamicTest("Visit second page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    }),
                    dynamicTest("Visit third page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    })
                )),
                dynamicStep("Log-out", () -> {
                    // visit logout URL
                    // assert user has been logged out
                })
        );
    }
@sormuras

This comment has been minimized.

Show comment
Hide comment
@sormuras

sormuras May 4, 2017

Member

I had some fun to implement a variation of my suggestion above. Assume your example scenario outputs:

  | '-- scenarioTests() [OK]
  |   +-- Visit page requiring authorization while not logged in [OK]
  |   +-- Log-in [OK]
  |   +-- Can access several pages while logged in [OK]
  |   | +-- Visit second page requiring authorization while logged in [OK]
  |   | '-- Visit third page requiring authorization while logged in [OK]
  |   '-- Log-out [OK]

Now, insert some failing check, like:

        dynamicTest("Log-in", () -> {
          fail("you shall not pass");
        }),

...and it takes the short way out:

  | +-- scenarioTests() [OK]
  | | +-- Visit page requiring authorization while not logged in [OK]
  | | '-- Log-in [X] you shall not pass

See https://github.com/junit-team/junit5/compare/issues/48-testfactory_failfast for an initial draft.

Member

sormuras commented May 4, 2017

I had some fun to implement a variation of my suggestion above. Assume your example scenario outputs:

  | '-- scenarioTests() [OK]
  |   +-- Visit page requiring authorization while not logged in [OK]
  |   +-- Log-in [OK]
  |   +-- Can access several pages while logged in [OK]
  |   | +-- Visit second page requiring authorization while logged in [OK]
  |   | '-- Visit third page requiring authorization while logged in [OK]
  |   '-- Log-out [OK]

Now, insert some failing check, like:

        dynamicTest("Log-in", () -> {
          fail("you shall not pass");
        }),

...and it takes the short way out:

  | +-- scenarioTests() [OK]
  | | +-- Visit page requiring authorization while not logged in [OK]
  | | '-- Log-in [X] you shall not pass

See https://github.com/junit-team/junit5/compare/issues/48-testfactory_failfast for an initial draft.

@sormuras

This comment has been minimized.

Show comment
Hide comment
@sormuras

sormuras May 4, 2017

Member

Meta-annotated Scenario annotation added: 9b1f57b ... looks nice, but feels like an overkill.

Member

sormuras commented May 4, 2017

Meta-annotated Scenario annotation added: 9b1f57b ... looks nice, but feels like an overkill.

@mfulton26

This comment has been minimized.

Show comment
Hide comment
@mfulton26

mfulton26 May 4, 2017

Neat.

One of the reasons I was thinking of being able to mix tests with steps (and containers with scenarios) was so that you can specify nonblocking tests (or containers) so that during a scenario you could specify some tests/containers that if they failed they wouldn't block execution of the remaining steps/scenarios.

As such, I do not believe that a single Scenario annotation is sufficient but a way to mix and match.

mfulton26 commented May 4, 2017

Neat.

One of the reasons I was thinking of being able to mix tests with steps (and containers with scenarios) was so that you can specify nonblocking tests (or containers) so that during a scenario you could specify some tests/containers that if they failed they wouldn't block execution of the remaining steps/scenarios.

As such, I do not believe that a single Scenario annotation is sufficient but a way to mix and match.

@sormuras

This comment has been minimized.

Show comment
Hide comment
@sormuras

sormuras May 4, 2017

Member

dynamicStep introduced. That simplified the implementation as well.

I left dynamicScenario out on purpose, because a container can not produce an execution result. If one its blocking nodes fails, the container fails too. If all nodes are executed successfully, it is marked successful too.

Member

sormuras commented May 4, 2017

dynamicStep introduced. That simplified the implementation as well.

I left dynamicScenario out on purpose, because a container can not produce an execution result. If one its blocking nodes fails, the container fails too. If all nodes are executed successfully, it is marked successful too.

@jillesvangurp

This comment has been minimized.

Show comment
Hide comment
@jillesvangurp

jillesvangurp May 5, 2017

@sormuras awesome; I like where this is going. Just occured to me that it would be nice if methods like dynamicContainer could accept a varargs of DynamicTest...subTests. There are a lot of calls to Stream.of in the example.

jillesvangurp commented May 5, 2017

@sormuras awesome; I like where this is going. Just occured to me that it would be nice if methods like dynamicContainer could accept a varargs of DynamicTest...subTests. There are a lot of calls to Stream.of in the example.

@jillesvangurp

This comment has been minimized.

Show comment
Hide comment
@jillesvangurp

jillesvangurp May 5, 2017

Likewise, it would save a bit of boilerplate and nesting if you could make the testFactory an annotated field instead of a method:

@TestFactory
Stream<DynamicNode> scenarioTests = dynamicScenario("Web Security Scenario", ....);

Just a thought; this might violate some notions people have about statefulness.

jillesvangurp commented May 5, 2017

Likewise, it would save a bit of boilerplate and nesting if you could make the testFactory an annotated field instead of a method:

@TestFactory
Stream<DynamicNode> scenarioTests = dynamicScenario("Web Security Scenario", ....);

Just a thought; this might violate some notions people have about statefulness.

@sormuras

This comment has been minimized.

Show comment
Hide comment
@sormuras

sormuras May 5, 2017

Member

Good idea, @jillesvangurp -- already added via 881a444

Hm, I don't like the "annotated field instead of a method" idea at all. Methods should stay the preferred entry points for Jupiter tests.

Member

sormuras commented May 5, 2017

Good idea, @jillesvangurp -- already added via 881a444

Hm, I don't like the "annotated field instead of a method" idea at all. Methods should stay the preferred entry points for Jupiter tests.

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen May 5, 2017

Member

@pgpx,

I guess I'm not sure why you would need/want each step to be a separate method (with state stored at a scenario level), and why before/after-step semantics are important (or couldn't be implemented with a simple lambda expression if needed).

Using lambda expressions and a builder pattern for implementing scenario tests is great for pure unit testing but not a good fit for complex integration and system testing.

For more complex use cases one needs the power of third-party (or self-built) extensions that tie into JUnit Jupiter's extension model and test execution lifecycle. With lambda expressions, however, that is not possible (unless you do something like this and implement #378).

Member

sbrannen commented May 5, 2017

@pgpx,

I guess I'm not sure why you would need/want each step to be a separate method (with state stored at a scenario level), and why before/after-step semantics are important (or couldn't be implemented with a simple lambda expression if needed).

Using lambda expressions and a builder pattern for implementing scenario tests is great for pure unit testing but not a good fit for complex integration and system testing.

For more complex use cases one needs the power of third-party (or self-built) extensions that tie into JUnit Jupiter's extension model and test execution lifecycle. With lambda expressions, however, that is not possible (unless you do something like this and implement #378).

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen May 5, 2017

Member

I can see that complex integration tests might require a lot of setup that you don't want to repeat, but that seems to be a different concern that could maybe be handled differently (e.g. look at how Spring's integration test framework caches its own context), and those tests are not really separate steps in a single scenario.

I'm very familiar with the Spring TestContext Framework (since I'm the author of said framework 😉), so let me provide some additional context here.

It's true that one does not want to needlessly repeat expensive setup, but that's only piece of the puzzle. The other piece of the puzzle is interaction with extensions between/around steps.

Although I did not state it in this issue's description, the WebSecurityScenarioTest example I provided is actually inspired from numerous "scenario testing" use cases that I have run into with Spring Boot, Spring Security, and the Spring TestContext Framework. Thus, in real life that example should look more like the following.

@ScenarioTest
@ExtendWith(SpringExtension.class)
@SpringBootTest(webEnvironment = MOCK)
@AutoConfigureMockMvc
@Transactional
class WebSecurityScenarioTest {

    @Autowired
    MockMvc mockMvc;

    @Step(next = "login")
    void visitPageRequiringAuthorizationWhileNotLoggedIn() {
        // attempt to visit page which requires that a user is logged in
        // assert user is redirected to login page
    }

    @Step(next = "visitSecondPageRequiringAuthorizationWhileLoggedIn")
    void login() {
        // submit login form with valid credentials
        // assert user is redirected back to previous page requiring authorization
    }

    @Step(next = "logout")
    @WithMockUser(roles = "USER")
    void visitSecondPageRequiringAuthorizationWhileLoggedIn() {
        // visit another page which requires that a user is logged in
        // assert user can access page
    }

    @Step(next = END)
    @WithMockUser(roles = "USER")
    void logout() {
        // visit logout URL
        // assert user has been logged out
    }

}

Furthermore, depending on the use case, individual methods may be annotated with @Commit, @Sql, etc. in addition to the @WithMockUser annotation (and similar annotations) from Spring Security.

My point: if the steps are not methods, it is impossible for the annotations (and therefore third-party extension features) to be applied at the step level.

In summary, although the lambda expression / builder / DSL being currently discussed looks super cool and powerful, it in fact (unfortunately) has several shortcomings.

Member

sbrannen commented May 5, 2017

I can see that complex integration tests might require a lot of setup that you don't want to repeat, but that seems to be a different concern that could maybe be handled differently (e.g. look at how Spring's integration test framework caches its own context), and those tests are not really separate steps in a single scenario.

I'm very familiar with the Spring TestContext Framework (since I'm the author of said framework 😉), so let me provide some additional context here.

It's true that one does not want to needlessly repeat expensive setup, but that's only piece of the puzzle. The other piece of the puzzle is interaction with extensions between/around steps.

Although I did not state it in this issue's description, the WebSecurityScenarioTest example I provided is actually inspired from numerous "scenario testing" use cases that I have run into with Spring Boot, Spring Security, and the Spring TestContext Framework. Thus, in real life that example should look more like the following.

@ScenarioTest
@ExtendWith(SpringExtension.class)
@SpringBootTest(webEnvironment = MOCK)
@AutoConfigureMockMvc
@Transactional
class WebSecurityScenarioTest {

    @Autowired
    MockMvc mockMvc;

    @Step(next = "login")
    void visitPageRequiringAuthorizationWhileNotLoggedIn() {
        // attempt to visit page which requires that a user is logged in
        // assert user is redirected to login page
    }

    @Step(next = "visitSecondPageRequiringAuthorizationWhileLoggedIn")
    void login() {
        // submit login form with valid credentials
        // assert user is redirected back to previous page requiring authorization
    }

    @Step(next = "logout")
    @WithMockUser(roles = "USER")
    void visitSecondPageRequiringAuthorizationWhileLoggedIn() {
        // visit another page which requires that a user is logged in
        // assert user can access page
    }

    @Step(next = END)
    @WithMockUser(roles = "USER")
    void logout() {
        // visit logout URL
        // assert user has been logged out
    }

}

Furthermore, depending on the use case, individual methods may be annotated with @Commit, @Sql, etc. in addition to the @WithMockUser annotation (and similar annotations) from Spring Security.

My point: if the steps are not methods, it is impossible for the annotations (and therefore third-party extension features) to be applied at the step level.

In summary, although the lambda expression / builder / DSL being currently discussed looks super cool and powerful, it in fact (unfortunately) has several shortcomings.

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen May 5, 2017

Member

And for those interested in seeing live, working examples...

  • TestNG: SimpleScenarioTestNgTests
  • JUnit 4: RestEventsControllerTests
    • RestEventsControllerTests is not actually a scenario test since it's not possible to implement such a test in JUnit 4, but it could be converted to a scenario test in JUnit Jupiter (if supported).
    • In other words, ideally one should be able to implement something analogous to SimpleScenarioTestNgTests in JUnit Jupiter. 😉
Member

sbrannen commented May 5, 2017

And for those interested in seeing live, working examples...

  • TestNG: SimpleScenarioTestNgTests
  • JUnit 4: RestEventsControllerTests
    • RestEventsControllerTests is not actually a scenario test since it's not possible to implement such a test in JUnit 4, but it could be converted to a scenario test in JUnit Jupiter (if supported).
    • In other words, ideally one should be able to implement something analogous to SimpleScenarioTestNgTests in JUnit Jupiter. 😉
@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen May 5, 2017

Member

Now, having said all that...

I am not saying that I am totally against some form of scenario tests based on lambda expressions and a DSL, but I think that is a separate topic that should be addressed independently.

Member

sbrannen commented May 5, 2017

Now, having said all that...

I am not saying that I am totally against some form of scenario tests based on lambda expressions and a DSL, but I think that is a separate topic that should be addressed independently.

@sormuras

This comment has been minimized.

Show comment
Hide comment
@sormuras

sormuras May 5, 2017

Member

In summary, although the lambda expression / builder / DSL being currently discussed looks super cool and powerful, it in fact (unfortunately) has several shortcomings.
[...]
I am not saying that I am totally against some form of scenario tests based on lambda expressions and a DSL, but I think that is a separate topic that should be addressed independently.

I totally agree, @sbrannen -- that's why I put the "does not solve" disclaimer at the top of #838

All further discussion of the "fail-fast DSL dynamic tests" feature should be made at #838

Member

sormuras commented May 5, 2017

In summary, although the lambda expression / builder / DSL being currently discussed looks super cool and powerful, it in fact (unfortunately) has several shortcomings.
[...]
I am not saying that I am totally against some form of scenario tests based on lambda expressions and a DSL, but I think that is a separate topic that should be addressed independently.

I totally agree, @sbrannen -- that's why I put the "does not solve" disclaimer at the top of #838

All further discussion of the "fail-fast DSL dynamic tests" feature should be made at #838

@mfulton26

This comment has been minimized.

Show comment
Hide comment
@mfulton26

mfulton26 May 5, 2017

One issue I see with @Step(next = "$methodName") is that there is no way to express that multiple tests follow this test but in no particular order (which would be especially useful in integration tests if concurrent execution is supported). TestNG takes the "dependsOn" route which seems to me currently to be a better route. Thoughts?

mfulton26 commented May 5, 2017

One issue I see with @Step(next = "$methodName") is that there is no way to express that multiple tests follow this test but in no particular order (which would be especially useful in integration tests if concurrent execution is supported). TestNG takes the "dependsOn" route which seems to me currently to be a better route. Thoughts?

@tnimni

This comment has been minimized.

Show comment
Hide comment
@tnimni

tnimni Nov 19, 2017

is there an implementation in junit 5.0.2 for test sequence?

tnimni commented Nov 19, 2017

is there an implementation in junit 5.0.2 for test sequence?

@marcphilipp

This comment has been minimized.

Show comment
Hide comment
@marcphilipp

marcphilipp Nov 19, 2017

Member

@tnimni No, this issue is still open and so is #13.

Member

marcphilipp commented Nov 19, 2017

@tnimni No, this issue is still open and so is #13.

@Jonarzz

This comment has been minimized.

Show comment
Hide comment
@Jonarzz

Jonarzz Mar 22, 2018

Is it known, which release will contain changes related to this issue?

Jonarzz commented Mar 22, 2018

Is it known, which release will contain changes related to this issue?

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Mar 22, 2018

Member

It is not "known", since we cannot predict the future.

But... this issue is currently assigned to the 5.2 Backlog. Thus, the intended release is 5.2.

FYI: you can always look at the assigned milestone (see information panel to the right of any issue description on GitHub) to infer this information on your own. 😉

Member

sbrannen commented Mar 22, 2018

It is not "known", since we cannot predict the future.

But... this issue is currently assigned to the 5.2 Backlog. Thus, the intended release is 5.2.

FYI: you can always look at the assigned milestone (see information panel to the right of any issue description on GitHub) to infer this information on your own. 😉

@mibutec

This comment has been minimized.

Show comment
Hide comment
@mibutec

mibutec Jun 8, 2018

In my company we are using a JS (i.e. Jasmine) inspired way of writing Scenarios.

@Scenario("Some succeeding scenario")
public void succeedingScenario() {
  Given("Some given condition", () -> {
    // some code
  });

  When("Some when condition", () -> {
    // some code	
  });

  Then("Some assertion" , () -> {
    // some code
  });
}

It lecks usage of JUnit default reporting, otherwise it has all the abilities of using extensions and all the abilities of java (decisions, loops, ...) to describe tests. Just as an idea.

mibutec commented Jun 8, 2018

In my company we are using a JS (i.e. Jasmine) inspired way of writing Scenarios.

@Scenario("Some succeeding scenario")
public void succeedingScenario() {
  Given("Some given condition", () -> {
    // some code
  });

  When("Some when condition", () -> {
    // some code	
  });

  Then("Some assertion" , () -> {
    // some code
  });
}

It lecks usage of JUnit default reporting, otherwise it has all the abilities of using extensions and all the abilities of java (decisions, loops, ...) to describe tests. Just as an idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment