Investigate more extensible solution for test resolution

[marcphilipp]

No description provided.

[schauder]

I wrote a mail to the mailing list last week, which among other things touched on the test resolution & execution process.

To which @bechte replied (https://groups.yahoo.com/neo/groups/junit/conversations/messages/24734)

The concept of a test engine covers both, lookup and execute tests. The core framework is not supposed to execute the tests itself, because a test engine could provide very specific test behavior, e.g. running cucumber tests, requiring a very specific execution path. Therefore, we decided that the engine is responsible for both. In case of the JUnit5TestEngine, we will provide extension points that can be used by extension providers to hook into the standard „Java testing“ test run circle. We are still working on this extension model, therefore, there is not much within the code, yet. Please be patient about this feature. I hope that we are able to provide it within the next iteration, too.

I still think that executing the tests should be done by some internal Java component, not some pluggable mechanism like a TestEngine.

Tools like IDEs will very soon rely heavily on the correct events/information delivered through some kind of listener API. If the TestEngines have to do this, because they are executing the tests, this will be broken for many TestEngines to come. But if it has to be the same all the time, it should be implemented only once.

I like the Cucumber example. I fully agree that it should be possible to execute write an Engine that registers tests as Cucumber does. And there is some challenging work in this, to find the right abstraction (like: What exactly is a Test, and do we need some Granularity above and below Tests.) But this needs to be represented in a TestSpecifcation, so JUnit can properly report on it. And in the end a Test just gets executed, succeeds, fails or gets skipped, there is not much any Engine can or should change about that.

And if TestEngine actually needs itself for the execution, there is no problem in having a Statement (using the Term from JUnit4 Rules) to reference and use the TestEngine it was created by.

Keep up the good work.

[jlink]

If I understand you right you'd prefer something along the following:

interface  ContainerDescriptor extends TestDescriptor {
    Executable beforeAll();
    Executable afterAll();
}

interface LeafDescriptor extends TestDescriptor {
    Executable runTest();
}

Then the launcher could execute the tests and make sure that the correct invocation of test execution events takes places.

Well worth a consideration IMO.

[schauder]

Yes.

[jlink]

Additional complexity: For discovery engines must be given an object / lambda with which they can do things like create dynamic tests, forward additional test data and like during test execution later.

I still think we should consider this approach and see if all types of engines that we think of would fit into such a model.

[jlink]

After some discussion with Marc (he did some concrete experimentation that suggests that parallelization would be a piece of cake with this approach) we feel the need to hand down stuff from a container to a child. This could be done via generics, for example instead of MutableTestDescriptor we could have two interfaces along those lines:

public interface ParentTestDescriptor<T> extends TestDescriptor {
    @Override
    Set<ChildTestDescriptor<T>> getChildren();
    void addChild(ChildTestDescriptor<T> descriptor);
    void removeChild(ChildTestDescriptor<T> descriptor);
    Callable<T> beforeAll();
    Consumer<Throwable> afterAll();
}

public interface ChildTestDescriptor<T> extends TestDescriptor {
    @Override
    Optional<ParentTestDescriptor<T>> getParent();
    void setParent(ParentTestDescriptor<T> parent);
    Consumer<T> runTest();
}

A specific engine could then implement type-safe classes. Intermediate containers would implement both interfaces, e.g. a nested test is both child and parent.

[marcphilipp]

I've just looked into JUnit 4 again. As much as I like the approach suggested here, I don't see how we could implement the JUnit 4 engine with it. 😢

[jlink]

I have an idea; maybe we can have two different flavours of engines. Let's
talk about it 2morrow.

2015-12-07 19:55 GMT+01:00 Marc Philipp notifications@github.com:

I've just looked into JUnit 4 again. As much as I like the approach
suggested here, I don't see how we could implement the JUnit 4 engine with
it. [image: 😢]

—
Reply to this email directly or view it on GitHub
#29 (comment)
.

[schauder]

I would like to understand what the problem is with JUnit 4 and this approach.