JUnit fails to report results of some tests if used with surefire and parallel option including methods

[apierzch]

Example: run mvn test in JUnitParams project with following configuration:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.18.1</version>

                <configuration>
                    <runOrder>filesystem</runOrder>
                    <parallel>classesAndMethods</parallel>
                    <perCoreThreadCount>false</perCoreThreadCount>
                    <threadCount>16</threadCount>
                    <useUnlimitedThreads>false</useUnlimitedThreads>
                    <threadCountSuites>0</threadCountSuites>
                    <threadCountClasses>0</threadCountClasses>
                    <threadCountMethods>0</threadCountMethods>
                    <parallelOptimized>true</parallelOptimized>
                    <includes>
                        <include>**/SubclassTest.java</include>
                        <include>**/Samples_*</include>
                    </includes>
                </configuration>
            </plugin>

This should give total of 44 tests as of today, but often it will give less than that. Contact me for details about research done so far.
Suspicious places:

• org.apache.maven.surefire.junitcore.JUnitCoreRunListener#fillTestCountMap
• org.apache.maven.surefire.junitcore.ClassesParallelRunListener#checkIfTestSetCanBeReported
• org.apache.maven.surefire.junitcore.TestSet#setAllScheduled

Currently following values of surefire "parallel" option should not be used:

• "classesAndMethods"
• "methods"
• "both"

[skyghis]

I did some research about this annoying bug.
With an empirical debugging/testing method I fixed the issue by removing the index from the Utils#uniqueMethodId. I found that the index is sometime not consistent during the run.

I suppose the index in the uniqueMethodId have a good reason to be here, but all the unit tests passes without it and my tests passes as well.

I let you try and test my method and review the causes.
Don't hesitate if you have more questions.

[dankirkd]

@apierzch and @skyghis - Has there been any advance on this issue and line of research?

[skyghis]

Sorry @dankirkd, I no longer use JUnitParams. I now use Junit5 with integrated parameterized tests.