JUnit5: StepListener get wrongly SUCCESS when an Exception is thrown in test

[zeners]

What happened?

net.serenitybdd.junit5.SerenityTestExecutionListener#testFinished missed to call updateResultsUsingTestExecutionResult() with TestExecutionResult.Status.FAILED because result.isLessSevereThan(TestResult.FAILURE) has to be negated or flipped.

What did you expect to happen?

get the correct FAILED result in net.thucydides.model.steps.StepListener#testFinished(net.thucydides.model.domain.TestOutcome) if an exception is thrown inside a test, but outside a step

Serenity BDD version

4.0.12

JDK version

17

Execution environment

caused by net.serenity-bdd:serenity-junit5:4.0.12

How to reproduce the bug.

• create a junit5 test with throw new RuntimeException("Test-Error");
• do one of
  • create a net.thucydides.model.steps.StepListener service and try to log result.getResult() inside void testFinished(TestOutcome result)
  • set a breakpoint in SerenityTestExecutionListener#testFinished
  • register net.thucydides.core.steps.Listeners#getLoggingListener() (eg with StepEventBus.getEventBus().registerListener(getLoggingListener()))
• run the test

test failed, but the Listener got an result SUCCESS

this can be seen

• inside testFinished(TestOutcome result) with result.getResult() == TestResult.SUCCESS
• that SerenityTestExecutionListener#testFinished missed to call updateResultsUsingTestExecutionResult()
• that the loggingListener prints Test Passed

please note that the method void testFailed(TestOutcome testOutcome, Throwable cause) is correctly called

How can we make it happen?

Add it to the Serenity BDD backlog and wait for a volunteer to pick it up

[wakaleo]

Can you propose a PR?

[wakaleo]

I would need to check but I believe if a test fails, the test finished listener is not meant to be triggered.

[wakaleo]

(Also 4.0.12 is an old version, have you tried with the latest versions)

[zeners]

• the test finished is called, if i call the missed routine inside debugger
• the spotted line is unchanged in current master

[wakaleo]

Can you propose a PR?

[zeners]

will take some time, my dev environment is outdated

[zeners]

i have troubles to reproduce it with current version 4.1.6

[zeners]

found:
4.0.12 > TestResult.SUCCESS.isLessSevereThan(TestResult.FAILURE) return false
4.1.6 > TestResult.SUCCESS.isLessSevereThan(TestResult.FAILURE) return true (correct)

[zeners]

duplicate of #3299