How to use internal custom TestEngine implementations in Maven-Surefire-Plugin?

[performrumpf]

For developing custom TestEngine classes there already is a lot of documentation on how to use them:

• ServiceLoader and plugin system
• LauncherConfig

Well, I tried to use a custom TestEngine with ServiceLoader (intern), it works fine in IntelliJ or using a Launcher, but I didn't get anything while running mvn test.

In Maven Surefire documentation it's mentioned that you only need to add JUnit-Jupiter-Engine in dependencies without the help of an provider. But these engines are external. What about internal engines?

[sormuras]

But these engines are external. What about internal engines?

What are external and internal engines?

Do you have a pom.xml snippet to share?

[sormuras]

Here is an integration test showing how to include three engines in a pom.xml:

https://github.com/apache/maven-surefire/blob/19006aa70f36705f399b8c105a16f636904f00f3/surefire-its/src/test/resources/junit-platform-multiple-engines/pom.xml#L36-L71

Basically, any test engine implementation available at test runtime (via class-path or module-path) that is registered via the ServiceLoader plugin system is picked up the the JUnit Platform.

[performrumpf]

    <properties>
        <java.version>1.8</java.version>
        <junit.platform.version>1.3.1</junit.platform.version>
    </properties>

    <build>
        <directory>target</directory>
        <outputDirectory>target/classes</outputDirectory>
        <testOutputDirectory>target/test-classes</testOutputDirectory>
        <sourceDirectory>src/main/java</sourceDirectory>
        <testSourceDirectory>src/test/java</testSourceDirectory>
        <resources>
            <resource>
                <directory>src/main/resources</directory>
            </resource>
        </resources>
        <testResources>
            <testResource>
                <directory>src/test/resources</directory>
            </testResource>
        </testResources>

        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.7.0</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-engine</artifactId>
            <version>${junit.platform.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-launcher</artifactId>
            <version>${junit.platform.version}</version>
        </dependency>
    </dependencies>

Project structure

• src
  • main
    • java
    • resources
  • test
    • java
      • com.experiment.engine
        • MyTestEngine.java
        • MyTestDescription.java
    • resources
      • META-INF
        • services
          • org.junit.platform.engine.TestEngine

[performrumpf]

I overwrote discover and execute methods in a way that engineExecutionListener fires "start" and "finish" for the root and one child.

Internal = same maven project
External = using in pom

[sormuras]

I overwrote discover and execute methods in a way that engineExecutionListener fires "start" and "finish" for the root and one child

Are you extending JupiterTestEngine? Or extending some class in junit-platform-launcher?

[performrumpf]

I don't extend from JupiterTestEngine as that test engine will build tests from other resources (i.e. XML files). I use launcher only to see if LauncherConfig works or (for the future) if DiscoverySelectors work properly.

public class MyTestEngine implements TestEngine {
   // implemented methods
}

[sormuras]

Can you please move your com.experiment.engine with its org.junit.platform.engine.TestEngine into the main source set and see if Surefire now sees your engine?

[sormuras]

What is listed in Surefire's class-paths when you invoke your build with mvn --debug ...?

[performrumpf]

unfortunately moving it to main didn't help. Also I couldn't find any useful information in debug mode about class path. The only line i could find with class path was
<additionalClasspathElements>${maven.test.additionalClasspath}</additionalClasspathElements>
<classpathDependencyExcludes>${maven.test.dependency.excludes}</classpathDependencyExcludes>

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< com.experimental:junit5-sandbox >-------------------
[INFO] Building junit5-sandbox 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ junit5-sandbox ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ junit5-sandbox ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ junit5-sandbox ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 34 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ junit5-sandbox ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is platform dependent!
[INFO] Compiling 1 source file to C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ junit5-sandbox ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.773 s
[INFO] Finished at: 2018-10-12T11:10:58+02:00
[INFO] ------------------------------------------------------------------------

[sbrannen]

What happens if you move META-INF/services/org.junit.platform.engine.TestEngine to src/main/resources as well?

[sormuras]

This is not a --debug enabled printout. Seems only [INFO]- and [DEBUG]-level loggings are emitted.

[performrumpf]

What happens if you move META-INF/services/org.junit.platform.engine.TestEngine to src/main/resources as well?

Already did, same result: nothing

[performrumpf]

Does this help you?

[DEBUG]   (s) additionalClasspathElements = []
[DEBUG]   (s) basedir = C:\Users\c.rumpf\Documents\perform\junit5sandbox
[DEBUG]   (s) childDelegation = false
[DEBUG]   (s) classesDirectory = C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\classes
[DEBUG]   (s) classpathDependencyExcludes = []
[DEBUG]   (s) dependenciesToScan = []
[DEBUG]   (s) disableXmlReport = false
[DEBUG]   (s) enableAssertions = true
[DEBUG]   (f) forkCount = 1
[DEBUG]   (s) forkMode = once
[DEBUG]   (s) forkedProcessExitTimeoutInSeconds = 30
[DEBUG]   (s) junitArtifactName = junit:junit
[DEBUG]   (s) junitPlatformArtifactName = org.junit.platform:junit-platform-engine
[DEBUG]   (s) localRepository =       id: local
      url: file:///C:/Users/c.rumpf/.m2/repository/
   layout: default
snapshots: [enabled => true, update => always]
 releases: [enabled => true, update => always]

[DEBUG]   (f) parallelMavenExecution = false
[DEBUG]   (s) parallelOptimized = true
[DEBUG]   (s) perCoreThreadCount = true

[sormuras]

Below those configurations there should be line like [DEBUG] -- end configuration -- and then the interesting lines start. Look for lines starting with [DEBUG] test(compact) classpath: and [DEBUG] provider(compact) classpath:.

[performrumpf]

At "maven-compiler-plugin":

[DEBUG] Classpath:
[DEBUG]  C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes
[DEBUG]  C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\classes
[DEBUG]  C:\Users\c.rumpf\.m2\repository\org\junit\platform\junit-platform-engine\1.3.1\junit-platform-engine-1.3.1.jar
[DEBUG]  C:\Users\c.rumpf\.m2\repository\org\apiguardian\apiguardian-api\1.0.0\apiguardian-api-1.0.0.jar
[DEBUG]  C:\Users\c.rumpf\.m2\repository\org\junit\platform\junit-platform-commons\1.3.1\junit-platform-commons-1.3.1.jar
[DEBUG]  C:\Users\c.rumpf\.m2\repository\org\opentest4j\opentest4j\1.1.1\opentest4j-1.1.1.jar
[DEBUG]  C:\Users\c.rumpf\.m2\repository\org\junit\platform\junit-platform-launcher\1.3.1\junit-platform-launcher-1.3.1.jar

Surefire didn't display any classpath informations

[sormuras]

Does C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes contain the MyEngine.class AND the META-INF/services/org.junit.platform.engine.TestEngine file?

Surefire didn't display any classpath informations

Strange...

[performrumpf]

Does C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes contain the MyEngine.class AND the META-INF/services/org.junit.platform.engine.TestEngine file?

When they are under src/test/..., then yes. Otherwise they are in target\classes

[sormuras]

Some more suggestions:

• Add <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> to your <properties>
• Update the maven-compiler-plugin to 3.8.0
• Help Surefire to select its surefire-junit-platform provider

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
                <dependencies>
                    <dependency>
                        <groupId>org.apache.maven.surefire</groupId>
                        <artifactId>surefire-junit-platform</artifactId>
                        <version>2.22.1</version>
                    </dependency>
                </dependencies>
            </plugin>

For details see https://maven.apache.org/surefire/maven-surefire-plugin/examples/providers.html

Got it working with the ice.cream engine -- without any src/main directory.

[performrumpf]

I made the suggested changes, but I don't get this last part of yours. Maybe Surefire is handling test discovery from a TestEngine differently then DefaultLauncher.

C:\Users\c.rumpf\Documents\perform\junit5sandbox>mvn test
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< com.experimental:junit5-sandbox >-------------------
[INFO] Building junit5-sandbox 1.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ junit5-sandbox ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ junit5-sandbox ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ junit5-sandbox ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 35 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ junit5-sandbox ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 4 source files to C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ junit5-sandbox ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.708 s
[INFO] Finished at: 2018-10-16T08:55:47+02:00
[INFO] ------------------------------------------------------------------------

Here my pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.experimental</groupId>
    <artifactId>junit5-sandbox</artifactId>
    <version>1.0.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>
        <junit.platform.version>1.3.1</junit.platform.version>
    </properties>

    <build>

        <directory>target</directory>
        <outputDirectory>target/classes</outputDirectory>
        <testOutputDirectory>target/test-classes</testOutputDirectory>
        <sourceDirectory>src/main/java</sourceDirectory>
        <testSourceDirectory>src/test/java</testSourceDirectory>
        <resources>
            <resource>
                <directory>src/main/resources</directory>
            </resource>
        </resources>
        <testResources>
            <testResource>
                <directory>src/test/resources</directory>
            </testResource>
        </testResources>

        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
                <dependencies>
                    <dependency>
                        <groupId>org.apache.maven.surefire</groupId>
                        <artifactId>surefire-junit-platform</artifactId>
                        <version>2.22.1</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-engine</artifactId>
            <version>${junit.platform.version}</version>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-launcher</artifactId>
            <version>${junit.platform.version}</version>
        </dependency>
    </dependencies>
</project>

And here is MyTestEngine.java

public class MyTestEngine implements TestEngine {

    @Override
    public String getId() {
        return "my-test-engine";
    }

    @Override
    public TestDescriptor discover(EngineDiscoveryRequest engineDiscoveryRequest, UniqueId uniqueId) {
        UniqueId rootId = uniqueId.append("container", "root#1");
        TestDescriptor root = new MyEngineDescriptor(uniqueId,"Root Description");
        TestDescriptor rootContainer = new MyTestDescriptor(rootId, "Container no. 1", TestDescriptor.Type.CONTAINER);

        rootContainer.addChild(new MyTestDescriptor(rootId.append("test", "child#1"), "Child no. 1", TestDescriptor.Type.TEST));
        rootContainer.addChild(new MyTestDescriptor(rootId.append("test", "child#2"), "Child no. 2", TestDescriptor.Type.TEST));
        root.addChild(rootContainer);
        return root;
    }

    @Override
    public void execute(ExecutionRequest executionRequest) {

        EngineExecutionListener listener = executionRequest.getEngineExecutionListener();
        TestDescriptor root = executionRequest.getRootTestDescriptor();

        execute(listener, root);
    }

    private void execute(EngineExecutionListener listener, TestDescriptor testDescriptor) {
        listener.executionStarted(testDescriptor);

        if(testDescriptor.isContainer()) {
            for(TestDescriptor child : testDescriptor.getChildren()) {
                execute(listener, child);
            }
        }

        listener.executionFinished(testDescriptor, TestExecutionResult.successful());
    }

    @Override
    public Optional<String> getGroupId() {
        return Optional.of("com.experimental");
    }

    @Override
    public Optional<String> getArtifactId() {
        return Optional.of("my-test-engine");
    }

    @Override
    public Optional<String> getVersion() {
        return Optional.of("1.0.0");
    }
}

[performrumpf]

I guess this is a maven issue and not an JUnit platform issue. I did following steps which resulted in executing my test:

• Added JUnit-Jupiter 5.3.1 to dependencies
• Wrote a Java JUnit test with a dummy test method (no assertions or output)

C:\Users\c.rumpf\Documents\perform\junit5sandbox>mvn test
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< com.experimental:junit5-sandbox >-------------------
[INFO] Building junit5-sandbox 1.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ junit5-sandbox ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ junit5-sandbox ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ junit5-sandbox ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 35 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ junit5-sandbox ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 5 source files to C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ junit5-sandbox ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running Container no. 1
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 s - in Container no. 1
[INFO] Running com.experimental.test.MyTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s - in com.experimental.test.MyTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.956 s
[INFO] Finished at: 2018-10-16T09:37:41+02:00
[INFO] ------------------------------------------------------------------------

Maybe Surefire only consideres other TestEngine implementations if it finds at least one from JUnit.

[performrumpf]

I'd like to close this ticket as this is definitely a feature of Surefire. Following reasons:

• Surefire looks for files having a certain pattern (see documentation)
• The filter can be overwritten with a simple order: mvn test -Dtest=*
• The filter can be overwritten by adding <test>*</test> in plugin configuration
• Not recommended: Using a Java class which follows default filter criteria without having any test methods (i.e. foo/DummyTest.java)

Maybe this should be mentioned in JUnit 5 User Guide Chapter Configuring Test Engines.

[sbrannen]

Maybe this should be mentioned in JUnit 5 User Guide Chapter Configuring Test Engines.

Maven's default patterns and filtering are mentioned in the User Guide: https://junit.org/junit5/docs/current/user-guide/#running-tests-build-maven-filter-test-class-names

That section also links to: https://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html

Are you saying the User Guide should do more than it already does in this regard?

[sbrannen]

I'd like to close this ticket as this is definitely a feature of Surefire.

OK. I'm closing it for you now.

[sormuras]

Surefire should select the correct Framework Provider automatically: having org.junit.platform:junit-platform-engine in your test runtime dependencies should trigger the selection of its surefire-junit-platform. As that didn't work out in my example, I added the surefire-junit-platform provider to the plugin dependencies, which forces this provider to be selected and executed.