New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reach out to Maven Surefire maintainers regarding JUnit Platform provider #31

Closed
marcphilipp opened this Issue Dec 2, 2015 · 22 comments

Comments

Projects
None yet
5 participants
@marcphilipp
Member

marcphilipp commented Dec 2, 2015

No description provided.

@marcphilipp marcphilipp self-assigned this Dec 2, 2015

@marcphilipp marcphilipp added this to the Alpha M1 milestone Dec 2, 2015

@marcphilipp marcphilipp changed the title from Reach out to Maven Surefire regarding plugin to Reach out to Maven Surefire maintainers regarding plugin Dec 3, 2015

@marcphilipp

This comment has been minimized.

Show comment
Hide comment
@marcphilipp

marcphilipp Dec 10, 2015

Member

I've opened an issue in the Maven Surefire JIRA regarding future support for JUnit 5 in Maven Surefire.

Member

marcphilipp commented Dec 10, 2015

I've opened an issue in the Maven Surefire JIRA regarding future support for JUnit 5 in Maven Surefire.

@Tibor17

This comment has been minimized.

Show comment
Hide comment
@Tibor17

Tibor17 Dec 14, 2015

@marcphilipp
Can you show me a prototype where the launcher executes JUnit4 tests on the top of JUnit5 engine?
Can we still interpret this code in JUnit5?

Request req = Request.classes( computer, classesToRun );
Runner runner = req.getRunner();
notifier.addFirstListener( listener );
runner.run( notifier );

Tibor17 commented Dec 14, 2015

@marcphilipp
Can you show me a prototype where the launcher executes JUnit4 tests on the top of JUnit5 engine?
Can we still interpret this code in JUnit5?

Request req = Request.classes( computer, classesToRun );
Runner runner = req.getRunner();
notifier.addFirstListener( listener );
runner.run( notifier );
@Tibor17

This comment has been minimized.

Show comment
Hide comment
@Tibor17

Tibor17 Dec 14, 2015

For tagging we can use Surefire's groups and ignored tests will have to be reported in the same way as now, right?

Groups and tags seem to be compliant.

Tibor17 commented Dec 14, 2015

For tagging we can use Surefire's groups and ignored tests will have to be reported in the same way as now, right?

Groups and tags seem to be compliant.

@marcphilipp

This comment has been minimized.

Show comment
Hide comment
@marcphilipp

marcphilipp Dec 15, 2015

Member

@Tibor17 Please take a look at https://github.com/junit-team/junit5-samples/tree/c9027db477d54ca7a78069eaa9bc9edd819f1235/junit5-maven-consumer

It demonstrates that our prototype of a SurefireProvider provider can run both JUnit 5 and JUnit 4 tests.

The code you've posted only works in JUnit 4. The rough equivalent using the new Launcher API would look like:

Launcher launcher = new Launcher();
launcher.registerTestExecutionListeners(listener);
launcher.execute(TestPlanSpecification.build(TestPlanSpecification.forClasses(classesToRun)));

Currently there is no equivalent of Computer in JUnit 5. Do you need it for parallelization?

Member

marcphilipp commented Dec 15, 2015

@Tibor17 Please take a look at https://github.com/junit-team/junit5-samples/tree/c9027db477d54ca7a78069eaa9bc9edd819f1235/junit5-maven-consumer

It demonstrates that our prototype of a SurefireProvider provider can run both JUnit 5 and JUnit 4 tests.

The code you've posted only works in JUnit 4. The rough equivalent using the new Launcher API would look like:

Launcher launcher = new Launcher();
launcher.registerTestExecutionListeners(listener);
launcher.execute(TestPlanSpecification.build(TestPlanSpecification.forClasses(classesToRun)));

Currently there is no equivalent of Computer in JUnit 5. Do you need it for parallelization?

@kcooney

This comment has been minimized.

Show comment
Hide comment
@kcooney

kcooney Dec 15, 2015

Member

To clarify, Request isn't going away. It just won't allow you to run JUnit5-style tests

Member

kcooney commented Dec 15, 2015

To clarify, Request isn't going away. It just won't allow you to run JUnit5-style tests

@marcphilipp

This comment has been minimized.

Show comment
Hide comment
@marcphilipp

marcphilipp Dec 15, 2015

Member

Well, Request isn't going away, but you won't need it to run JUnit 4 or JUnit 5 tests. Instead you have to build a TestPlanSpecification that the new junit4-engine will convert into one or multiple Requests.

Member

marcphilipp commented Dec 15, 2015

Well, Request isn't going away, but you won't need it to run JUnit 4 or JUnit 5 tests. Instead you have to build a TestPlanSpecification that the new junit4-engine will convert into one or multiple Requests.

@marcphilipp

This comment has been minimized.

Show comment
Hide comment
@marcphilipp

This comment has been minimized.

Show comment
Hide comment
@marcphilipp

marcphilipp Dec 15, 2015

Member

@Tibor17 Here's an overview of our intended overall architecture:

junit

Member

marcphilipp commented Dec 15, 2015

@Tibor17 Here's an overview of our intended overall architecture:

junit

@Tibor17

This comment has been minimized.

Show comment
Hide comment
@Tibor17

Tibor17 Dec 15, 2015

IIUC Junit 4 should be wrapped in an engine and TestEngine interface should
be implemented. AFAIK the Launcher executes all tests over all engines. It
seems filtering of applicable classes is a subject to the implementation of
the particular engine, right?
Is the JUnit4TestEngine implementation complete?
The rule in Maven group is to wait for complete JUnit release and then
adapt Surefire to it in master. Meanwhile we can of course make a try and
create a branch with a new provider.

On Tue, Dec 15, 2015 at 6:07 PM, Marc Philipp notifications@github.com
wrote:

@Tibor17 https://github.com/Tibor17 Here's an overview of our intended
overall architecture:

[image: junit]
https://cloud.githubusercontent.com/assets/214207/11817699/68482a30-a356-11e5-8c38-c93d95988488.png


Reply to this email directly or view it on GitHub
#31 (comment)
.

Cheers
Tibor

Tibor17 commented Dec 15, 2015

IIUC Junit 4 should be wrapped in an engine and TestEngine interface should
be implemented. AFAIK the Launcher executes all tests over all engines. It
seems filtering of applicable classes is a subject to the implementation of
the particular engine, right?
Is the JUnit4TestEngine implementation complete?
The rule in Maven group is to wait for complete JUnit release and then
adapt Surefire to it in master. Meanwhile we can of course make a try and
create a branch with a new provider.

On Tue, Dec 15, 2015 at 6:07 PM, Marc Philipp notifications@github.com
wrote:

@Tibor17 https://github.com/Tibor17 Here's an overview of our intended
overall architecture:

[image: junit]
https://cloud.githubusercontent.com/assets/214207/11817699/68482a30-a356-11e5-8c38-c93d95988488.png


Reply to this email directly or view it on GitHub
#31 (comment)
.

Cheers
Tibor

@marcphilipp

This comment has been minimized.

Show comment
Hide comment
@marcphilipp

marcphilipp Dec 15, 2015

Member

Yes, exactly. We will provide engines for JUnit 4 and JUnit 5. However, their implementations are not complete, yet. Actually, we've deleted all the code from the prototype and started from scratch.

Our plan is to have a complete implementation of both engines and a stable API in an alpha release in February. I think it would be a good idea to then start a new provider in a branch. This way we'd stumble upon potential problems as early as possible.

@Tibor17 Does that make sense to you?

Member

marcphilipp commented Dec 15, 2015

Yes, exactly. We will provide engines for JUnit 4 and JUnit 5. However, their implementations are not complete, yet. Actually, we've deleted all the code from the prototype and started from scratch.

Our plan is to have a complete implementation of both engines and a stable API in an alpha release in February. I think it would be a good idea to then start a new provider in a branch. This way we'd stumble upon potential problems as early as possible.

@Tibor17 Does that make sense to you?

@Tibor17

This comment has been minimized.

Show comment
Hide comment
@Tibor17

Tibor17 Dec 15, 2015

Enjoy it :)
We, actually I, have a big task in Surefire. We have an ambition in
Surefire project to open our API for public use. The problem is that JIRA
users report controversial issue however both are valid for relevant use
cases. That's the reason why I need to design such abstraction which
satisfies the users and they would write their own extensions. I should
combine idea of your engines but I am not sure about complete picture of
the extensions yet. So my situation is quite complicated. Some people say
that generic API would never exist , I agree, some people say to use
traditional Maven artifacts as extensions and some people like scripting;
and I want all.

On Tue, Dec 15, 2015 at 8:51 PM, Marc Philipp notifications@github.com
wrote:

Yes, exactly. We will provide engines for JUnit 4 and JUnit 5. However,
their implementations are not complete, yet. Actually, we've deleted all
the code from the prototype and started from scratch.

Our plan is to have a complete implementation of both engines and a stable
API in an alpha release in February. I think it would be a good idea to
then start a new provider in a branch. This way we'd stumble upon potential
problems as early as possible.

@Tibor17 https://github.com/Tibor17 Does that make sense to you?


Reply to this email directly or view it on GitHub
#31 (comment)
.

Cheers
Tibor

Tibor17 commented Dec 15, 2015

Enjoy it :)
We, actually I, have a big task in Surefire. We have an ambition in
Surefire project to open our API for public use. The problem is that JIRA
users report controversial issue however both are valid for relevant use
cases. That's the reason why I need to design such abstraction which
satisfies the users and they would write their own extensions. I should
combine idea of your engines but I am not sure about complete picture of
the extensions yet. So my situation is quite complicated. Some people say
that generic API would never exist , I agree, some people say to use
traditional Maven artifacts as extensions and some people like scripting;
and I want all.

On Tue, Dec 15, 2015 at 8:51 PM, Marc Philipp notifications@github.com
wrote:

Yes, exactly. We will provide engines for JUnit 4 and JUnit 5. However,
their implementations are not complete, yet. Actually, we've deleted all
the code from the prototype and started from scratch.

Our plan is to have a complete implementation of both engines and a stable
API in an alpha release in February. I think it would be a good idea to
then start a new provider in a branch. This way we'd stumble upon potential
problems as early as possible.

@Tibor17 https://github.com/Tibor17 Does that make sense to you?


Reply to this email directly or view it on GitHub
#31 (comment)
.

Cheers
Tibor

@Tibor17

This comment has been minimized.

Show comment
Hide comment
@Tibor17

Tibor17 Dec 15, 2015

Question. Can you create such API where lazily loaded tests set would be
discovered on the fly?
Not sure if you have it already or not.
This means the length of the Set of classes is unknown ahead. This would
enable us to stream the tests from another machine and distribute the
classes via socket.
I think you can utilize Java 8 streams. WDYT?
In JUnit 4 it was not possible and therefore we complicated implementation
and always had to switch the logic.
I think you can always filter the particular class on the fly and you do
not need to have full test set ready to know if particular class has one
method to run.

On Tue, Dec 15, 2015 at 10:18 PM, Tibor Digana tibor.digana@googlemail.com
wrote:

Enjoy it :)
We, actually I, have a big task in Surefire. We have an ambition in
Surefire project to open our API for public use. The problem is that JIRA
users report controversial issue however both are valid for relevant use
cases. That's the reason why I need to design such abstraction which
satisfies the users and they would write their own extensions. I should
combine idea of your engines but I am not sure about complete picture of
the extensions yet. So my situation is quite complicated. Some people say
that generic API would never exist , I agree, some people say to use
traditional Maven artifacts as extensions and some people like scripting;
and I want all.

On Tue, Dec 15, 2015 at 8:51 PM, Marc Philipp notifications@github.com
wrote:

Yes, exactly. We will provide engines for JUnit 4 and JUnit 5. However,
their implementations are not complete, yet. Actually, we've deleted all
the code from the prototype and started from scratch.

Our plan is to have a complete implementation of both engines and a
stable API in an alpha release in February. I think it would be a good idea
to then start a new provider in a branch. This way we'd stumble upon
potential problems as early as possible.

@Tibor17 https://github.com/Tibor17 Does that make sense to you?


Reply to this email directly or view it on GitHub
#31 (comment)
.

Cheers
Tibor

Cheers
Tibor

Tibor17 commented Dec 15, 2015

Question. Can you create such API where lazily loaded tests set would be
discovered on the fly?
Not sure if you have it already or not.
This means the length of the Set of classes is unknown ahead. This would
enable us to stream the tests from another machine and distribute the
classes via socket.
I think you can utilize Java 8 streams. WDYT?
In JUnit 4 it was not possible and therefore we complicated implementation
and always had to switch the logic.
I think you can always filter the particular class on the fly and you do
not need to have full test set ready to know if particular class has one
method to run.

On Tue, Dec 15, 2015 at 10:18 PM, Tibor Digana tibor.digana@googlemail.com
wrote:

Enjoy it :)
We, actually I, have a big task in Surefire. We have an ambition in
Surefire project to open our API for public use. The problem is that JIRA
users report controversial issue however both are valid for relevant use
cases. That's the reason why I need to design such abstraction which
satisfies the users and they would write their own extensions. I should
combine idea of your engines but I am not sure about complete picture of
the extensions yet. So my situation is quite complicated. Some people say
that generic API would never exist , I agree, some people say to use
traditional Maven artifacts as extensions and some people like scripting;
and I want all.

On Tue, Dec 15, 2015 at 8:51 PM, Marc Philipp notifications@github.com
wrote:

Yes, exactly. We will provide engines for JUnit 4 and JUnit 5. However,
their implementations are not complete, yet. Actually, we've deleted all
the code from the prototype and started from scratch.

Our plan is to have a complete implementation of both engines and a
stable API in an alpha release in February. I think it would be a good idea
to then start a new provider in a branch. This way we'd stumble upon
potential problems as early as possible.

@Tibor17 https://github.com/Tibor17 Does that make sense to you?


Reply to this email directly or view it on GitHub
#31 (comment)
.

Cheers
Tibor

Cheers
Tibor

@marcphilipp

This comment has been minimized.

Show comment
Hide comment
@marcphilipp

marcphilipp Dec 17, 2015

Member

I'm not sure I understand you correctly.

Here's what I think you're up to:

  • In the VM where the Maven process is running you have a stream of classes you would like to run.
  • You want to dispatch the execution of those test classes one-by-one to a set of other VMs running on the same machine.

I assumed that was about what forking did in Surefire, already?

We have an API that allows TestEngines to discover "dynamic" tests at runtime: TestExecutionListener#dynamicTestRegistered(TestIdentifier). We plan to provide support for lambda-based testing, parameterized tests, theories/quickcheck this way.

Member

marcphilipp commented Dec 17, 2015

I'm not sure I understand you correctly.

Here's what I think you're up to:

  • In the VM where the Maven process is running you have a stream of classes you would like to run.
  • You want to dispatch the execution of those test classes one-by-one to a set of other VMs running on the same machine.

I assumed that was about what forking did in Surefire, already?

We have an API that allows TestEngines to discover "dynamic" tests at runtime: TestExecutionListener#dynamicTestRegistered(TestIdentifier). We plan to provide support for lambda-based testing, parameterized tests, theories/quickcheck this way.

@Tibor17

This comment has been minimized.

Show comment
Hide comment
@Tibor17

Tibor17 Dec 17, 2015

In the old JUnit4 we always created new JUnitCore and run only single class.
This is of course an overhead if you consider many fast running methods.
Of course we are streaming classes to the forked VM and we do NOT want
JUnit do that.
All we want to have is some kind of iterator used internally by JUnit5 and
we will provide it to JUnit5 from outside. We already have such one and the
methods like hasNext(), next() have a lock waiting for new stream data -
our goal.
The advantage against traditional JUnit is the fact that all the SPI etc.
was already started up and you do not need to filter all classes in the
begining. This will speed up the execution because you will filter only
particular class right now retrieved from Iterator#next().

On Thu, Dec 17, 2015 at 8:57 AM, Marc Philipp notifications@github.com
wrote:

I'm not sure I understand you correctly.

Here's what I think you're up to:

  • In the VM where the Maven process is running you have a stream of
    classes you would like to run.
  • You want to dispatch the execution of those test classes one-by-one
    to a set of other VMs running on the same machine.

I assumed that was about what forking did in Surefire, already?

We have an API that allows TestEngines to discover "dynamic" tests at
runtime: TestExecutionListener#dynamicTestRegistered(TestIdentifier). We
plan to provide support for lambda-based testing, parameterized tests,
theories/quickcheck this way.


Reply to this email directly or view it on GitHub
#31 (comment)
.

Cheers
Tibor

Tibor17 commented Dec 17, 2015

In the old JUnit4 we always created new JUnitCore and run only single class.
This is of course an overhead if you consider many fast running methods.
Of course we are streaming classes to the forked VM and we do NOT want
JUnit do that.
All we want to have is some kind of iterator used internally by JUnit5 and
we will provide it to JUnit5 from outside. We already have such one and the
methods like hasNext(), next() have a lock waiting for new stream data -
our goal.
The advantage against traditional JUnit is the fact that all the SPI etc.
was already started up and you do not need to filter all classes in the
begining. This will speed up the execution because you will filter only
particular class right now retrieved from Iterator#next().

On Thu, Dec 17, 2015 at 8:57 AM, Marc Philipp notifications@github.com
wrote:

I'm not sure I understand you correctly.

Here's what I think you're up to:

  • In the VM where the Maven process is running you have a stream of
    classes you would like to run.
  • You want to dispatch the execution of those test classes one-by-one
    to a set of other VMs running on the same machine.

I assumed that was about what forking did in Surefire, already?

We have an API that allows TestEngines to discover "dynamic" tests at
runtime: TestExecutionListener#dynamicTestRegistered(TestIdentifier). We
plan to provide support for lambda-based testing, parameterized tests,
theories/quickcheck this way.


Reply to this email directly or view it on GitHub
#31 (comment)
.

Cheers
Tibor

@Tibor17

This comment has been minimized.

Show comment
Hide comment
@Tibor17

Tibor17 Dec 19, 2015

@marcphilipp
Question. Is it possible to use individual ClassLoaders per test method in JUnit5?
Last time I did the same with JUnit4 and it was pain because I could not fully reuse BlockJUnitClassRunner. The rules handler was written with Reflection because the Runner class was loaded by System ClassLoader and the test class in another one.

Tibor17 commented Dec 19, 2015

@marcphilipp
Question. Is it possible to use individual ClassLoaders per test method in JUnit5?
Last time I did the same with JUnit4 and it was pain because I could not fully reuse BlockJUnitClassRunner. The rules handler was written with Reflection because the Runner class was loaded by System ClassLoader and the test class in another one.

@marcphilipp

This comment has been minimized.

Show comment
Hide comment
@marcphilipp

marcphilipp Dec 22, 2015

Member

@Tibor17 Individual class loaders per test method are not on our agenda at the moment. Can you please provide a few details about the use case we're talking about here?

Member

marcphilipp commented Dec 22, 2015

@Tibor17 Individual class loaders per test method are not on our agenda at the moment. Can you please provide a few details about the use case we're talking about here?

@Tibor17

This comment has been minimized.

Show comment
Hide comment
@Tibor17

Tibor17 Dec 23, 2015

yes I know it is not in your agenda, but for you maybe a little change in
abstraction would mean a big change for the world :)
Simply there were two use cases.
First use case needed to run parallel tests with a singleton. The status of
the singleton should not be shared across multiple tests.
Second use case was with Java EE CDI container to isolate the beans context
across parallel tests.

On Tue, Dec 22, 2015 at 1:55 PM, Marc Philipp notifications@github.com
wrote:

@Tibor17 https://github.com/Tibor17 Individual class loaders per test
method are not on our agenda at the moment. Can you please provide a few
details about the use case we're talking about here?


Reply to this email directly or view it on GitHub
#31 (comment)
.

Cheers
Tibor

Tibor17 commented Dec 23, 2015

yes I know it is not in your agenda, but for you maybe a little change in
abstraction would mean a big change for the world :)
Simply there were two use cases.
First use case needed to run parallel tests with a singleton. The status of
the singleton should not be shared across multiple tests.
Second use case was with Java EE CDI container to isolate the beans context
across parallel tests.

On Tue, Dec 22, 2015 at 1:55 PM, Marc Philipp notifications@github.com
wrote:

@Tibor17 https://github.com/Tibor17 Individual class loaders per test
method are not on our agenda at the moment. Can you please provide a few
details about the use case we're talking about here?


Reply to this email directly or view it on GitHub
#31 (comment)
.

Cheers
Tibor

@marcphilipp marcphilipp modified the milestones: M1, Alpha 1 Jan 11, 2016

@marcphilipp marcphilipp modified the milestones: 5.0 M2, 5.0 M1 Apr 15, 2016

@sbrannen sbrannen changed the title from Reach out to Maven Surefire maintainers regarding plugin to Reach out to Maven Surefire maintainers regarding JUnit Platform provider Jul 5, 2016

@sbrannen sbrannen modified the milestones: 5.0 M2, 5.0 M3 Jul 15, 2016

@marcphilipp

This comment has been minimized.

Show comment
Hide comment
@marcphilipp

marcphilipp Sep 2, 2016

Member

@britter is currently reaching out to the Maven Surefire developers.

Member

marcphilipp commented Sep 2, 2016

@britter is currently reaching out to the Maven Surefire developers.

@britter

This comment has been minimized.

Show comment
Hide comment
@britter

britter Sep 3, 2016

Contributor

You can follow the discussion here. Currently we're planning to meet on freenode to decide on how to move this issue forward.

Contributor

britter commented Sep 3, 2016

You can follow the discussion here. Currently we're planning to meet on freenode to decide on how to move this issue forward.

@sbrannen

This comment has been minimized.

Show comment
Hide comment
@sbrannen

sbrannen Sep 3, 2016

Member

Sounds good, @britter. Thanks for taking the initiative!

Member

sbrannen commented Sep 3, 2016

Sounds good, @britter. Thanks for taking the initiative!

@britter

This comment has been minimized.

Show comment
Hide comment
@britter

britter Sep 5, 2016

Contributor

Marc and I talked to Tibor via IRC yesterday. The Maven team wants to be sure the JUnit 5 surefire provider has all the features the old provider has. For this reason we're going to setup up their integration tests for JUnit 5. This also relates to my comment in #485 about maven groups vs. JUnit tags.

Contributor

britter commented Sep 5, 2016

Marc and I talked to Tibor via IRC yesterday. The Maven team wants to be sure the JUnit 5 surefire provider has all the features the old provider has. For this reason we're going to setup up their integration tests for JUnit 5. This also relates to my comment in #485 about maven groups vs. JUnit tags.

@marcphilipp marcphilipp modified the milestones: 5.0 M3, 5.0 Backlog Oct 11, 2016

@marcphilipp marcphilipp modified the milestones: 5.1+ Backlog, 5.0 M5 Dec 22, 2016

@marcphilipp marcphilipp modified the milestones: 5.0 M4, 5.0 M5 Mar 12, 2017

@marcphilipp

This comment has been minimized.

Show comment
Hide comment
@marcphilipp

marcphilipp Mar 12, 2017

Member

The initial contribution is done. Thus, I'm closing this issue.

Member

marcphilipp commented Mar 12, 2017

The initial contribution is done. Thus, I'm closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment