Exception thrown in @MethodSource method results in "no tests"

[ctron]

Steps to reproduce

Running the following code:

import java.util.List;

import org.junit.jupiter.api.TestInstance;
import org.junit.jupiter.api.TestInstance.Lifecycle;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;

@TestInstance(Lifecycle.PER_CLASS)
public class MethodSourceErrorTest {

    public List<String> getValues() {
        throw new RuntimeException("Foo"); // fails
        // return Arrays.asList("a"); // works
    }

    @ParameterizedTest
    @MethodSource("getValues")
    public void test1(String value) {
        System.out.println(value);
    }

}

Results in:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running foo.MethodSourceErrorTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 s - in foo.MethodSourceErrorTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

My expectation would be that:

• JUnit reports an error for the tests using the method source
• The build fails

Additional information:

• It fails the same way, being a static method

• It fails the same way, running the test from inside Eclipse:

  Screenshot of failure in Eclipse

  

Context

• Used versions (Jupiter/Vintage/Platform): 5.6.2, 1.6.2
• Build Tool/IDE: Eclipse 2020-06, Maven 3.6.3

Deliverables

• ...

[sbrannen]

It fails the same way, running the test from inside Eclipse:

Well, in Eclipse at least the exception is reported in the UI.

Look at the lower left corner or your screen shot.

The java.lang.RuntimeException: Foo is clearly present there as the cause of the failure.

Thus, if you want Eclipse to report the failure differently, you will need to open an ticket in the Eclipse bug tracker.

I assume the same is true for Maven Surefire and that the JUnit Platform reports the exception for the @MethodSource factory method (otherwise, Eclipse wouldn't have access to it to display it).

If you find that JUnit Jupiter does not properly report the failure to a TestExecutionListener registered in the JUnit Platform, please let us know.

[ctron]

Thanks for the explanation. I created a TestExecutionListener and dumped the result:

public class MyTestExecutionListener implements TestExecutionListener {

    public MyTestExecutionListener() {
        System.out.println("Here");
    }

    @Override
    public void executionFinished(TestIdentifier testIdentifier, TestExecutionResult testExecutionResult) {
        System.out.format("finished - id: %s, result: %s%n", testIdentifier, testExecutionResult);
    }
}

Which shows:

[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ bar ---
[WARNING] useSystemClassloader setting has no effect when not forking
Here
[INFO] Running foo.MethodSourceErrorTest
finished - id: TestIdentifier [uniqueId = '[engine:junit-jupiter]/[class:foo.MethodSourceErrorTest]/[test-template:test1(java.lang.String)]', parentId = '[engine:junit-jupiter]/[class:foo.MethodSourceErrorTest]', displayName = 'test1(String)', legacyReportingName = 'test1(String)', source = MethodSource [className = 'foo.MethodSourceErrorTest', methodName = 'test1', methodParameterTypes = 'java.lang.String'], tags = [], type = CONTAINER], result: TestExecutionResult [status = FAILED, throwable = java.lang.RuntimeException: Foo]
finished - id: TestIdentifier [uniqueId = '[engine:junit-jupiter]/[class:foo.MethodSourceErrorTest]', parentId = '[engine:junit-jupiter]', displayName = 'MethodSourceErrorTest', legacyReportingName = 'foo.MethodSourceErrorTest', source = ClassSource [className = 'foo.MethodSourceErrorTest', filePosition = null], tags = [], type = CONTAINER], result: TestExecutionResult [status = SUCCESSFUL, throwable = null]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 s - in foo.MethodSourceErrorTest
finished - id: TestIdentifier [uniqueId = '[engine:junit-jupiter]', parentId = null, displayName = 'JUnit Jupiter', legacyReportingName = 'JUnit Jupiter', source = null, tags = [], type = CONTAINER], result: TestExecutionResult [status = SUCCESSFUL, throwable = null]
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

So to me that looks like the exception gets reported, and Surefire doesn't record it properly. I raised an issue in their JIRA: https://issues.apache.org/jira/browse/SUREFIRE-1799

[sbrannen]

Thanks for verifying the status quo on your end with a custom TestExecutionListener and for raising the issue with Surefire.

In light of that, I'm closing this issue.