Nested Steps Without SerenityReporterParallel plugin Incorrectly Pass Tests

[mkonir0]

What happened?

When using nested steps in Serenity 4 with Cucumber and simultaneously not using the io.cucumber.core.plugin.SerenityReporterParallel plugin, tests that are expected to fail are incorrectly marked as passed.
This issue started occurring only after upgrading from Serenity 3 to Serenity 4 ( it worked fine before). I would also like to mention that we do not want to run steps in parallel, so using the SerenityReporterParallel plugin is not an option for us.

What did you expect to happen?

since nested step contains failing assertion tests should end up as failure.

Serenity BDD version

4.0.15 (same issue is in 4.0.21)

JDK version

17

Execution environment

<serenity.version>4.0.15
<junit-vintage-engine.version>5.1.0
<junit-platform-suite.version>1.10.0
<cucumber-junit-platform-engine.version>7.14.0

How to reproduce the bug.

To demonstrate the issue, I have created a fork of the serenity-cucumber-starter->
https://github.com/mkonir0/serenity-cucumber-starter
this commit demonstrates mentioned issue ->
mkonir0/serenity-cucumber-starter@dce0579

I have following feature file

Feature: Search by keyword

  @green
  Scenario: Searching for 'green'
    Then assert in double nested step method

and following stepdefinition

public class SearchStepDefinitions {
    @Steps
    private UserSteps userSteps;

    @Then("assert in double nested step method")
    public void assertResponseIsCustomHttpsBlockPage() {
        userSteps.assertInDoubleNestedStepMethod();
    }
}

and following steps

public class UserSteps {

	@Step
	public void assertInDoubleNestedStepMethod() {
		assertInStepMethod();
	}

	@Step
	public void assertInStepMethod() {
		assertThat(false).isTrue();
	}
}

and following CucumberTestSuite class

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("/features")
@ConfigurationParameter(key = PLUGIN_PROPERTY_NAME, value = "pretty,timeline:build/test-results/timeline")
public class CucumberTestSuite {
}

when I execute mvn verify goal without using the io.cucumber.core.plugin.SerenityReporterParallel plugin the tests falsly end up as success

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running starter.CucumberTestSuite

@green
Scenario: Searching for 'green'            # features/search/search_by_keyword.feature:4
10:40:20.563 [ForkJoinPool-2-worker-1] INFO  n.thucydides.core.steps.StepEventBus - Loading JUnit Platform configuration parameters from classpath resource [junit-platform.properties].
  Then assert in double nested step method # starter.stepdefinitions.SearchStepDefinitions.assertResponseIsCustomHttpsBlockPage()
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.582 s -- in starter.CucumberTestSuite

when I execute mvn verify goal with using the io.cucumber.core.plugin.SerenityReporterParallel plugin the tests end up as failure

Searching for 'green'(search-by-keyword;searching-for--green-)
--------------------------------------------------------------------------------
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.174 s <<< FAILURE! -- in starter.CucumberTestSuite
[ERROR] Search by keyword.Searching for 'green' -- Time elapsed: 1.122 s <<< FAILURE!
org.opentest4j.AssertionFailedError: 

Expecting value to be true but was false
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at starter.steps.UserSteps.assertInStepMethod(UserSteps.java:17)
        at starter.steps.UserSteps$ByteBuddy$caQJP6uq.assertInStepMethod$accessor$01Qi3QF7(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at net.thucydides.core.steps.StepInterceptor.invokeMethod(StepInterceptor.java:519)
        at net.thucydides.core.steps.StepInterceptor.executeTestStepMethod(StepInterceptor.java:499)
        at net.thucydides.core.steps.StepInterceptor.runTestStep(StepInterceptor.java:471)
        at net.thucydides.core.steps.StepInterceptor.runOrSkipMethod(StepInterceptor.java:213)
        at net.thucydides.core.steps.StepInterceptor.testStepResult(StepInterceptor.java:200)
        at net.thucydides.core.steps.StepInterceptor.intercept(StepInterceptor.java:92)
        at net.thucydides.core.steps.ProxyConfiguration$InterceptorDispatcher.intercept(ProxyConfiguration.java:51)
        at starter.steps.UserSteps$ByteBuddy$caQJP6uq.assertInStepMethod(Unknown Source)
        at starter.steps.UserSteps.assertInDoubleNestedStepMethod(UserSteps.java:12)
        at starter.steps.UserSteps$ByteBuddy$caQJP6uq.assertInDoubleNestedStepMethod$accessor$01Qi3QF7(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at net.thucydides.core.steps.StepInterceptor.invokeMethod(StepInterceptor.java:519)
        at net.thucydides.core.steps.StepInterceptor.executeTestStepMethod(StepInterceptor.java:499)
        at net.thucydides.core.steps.StepInterceptor.runTestStep(StepInterceptor.java:471)
        at net.thucydides.core.steps.StepInterceptor.runOrSkipMethod(StepInterceptor.java:213)
        at net.thucydides.core.steps.StepInterceptor.testStepResult(StepInterceptor.java:200)
        at net.thucydides.core.steps.StepInterceptor.intercept(StepInterceptor.java:92)
        at net.thucydides.core.steps.ProxyConfiguration$InterceptorDispatcher.intercept(ProxyConfiguration.java:51)
        at starter.steps.UserSteps$ByteBuddy$caQJP6uq.assertInDoubleNestedStepMethod(Unknown Source)
        at starter.stepdefinitions.SearchStepDefinitions.assertResponseIsCustomHttpsBlockPage(SearchStepDefinitions.java:21)
        at ✽.assert in double nested step method(classpath:features/search/search_by_keyword.feature:5)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   
Expecting value to be true but was false
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

How can we make it happen?

Work on this myself and propose a PR (with Serenity BDD team guidance)

[wakaleo]

You need to use this plugin for Serenity 4 and Cucumber 7 regardless of whether you are running the tests in parallel or not.

[mkonir0]

@wakaleo Thanks for help! I didn't notice it in any of the migration guides.

Also in one of our projects, where we're still using junit4
we have a single cucumber runner (which is beiing executed in runtime) -

import org.junit.runner.RunWith;
import io.cucumber.junit.CucumberOptions;
import net.serenitybdd.cucumber.CucumberWithSerenity;

@RunWith(CucumberWithSerenity.class)
@CucumberOptions(
		features = { ... },
		glue = { ... },
		plugin = { "com.oracle.runtime.cucumber.CustomCucumberListener", "io.cucumber.core.plugin.SerenityReporterParallel" })
public class CustomCucumberRunner {
}

when we upgraded to serenity4 and started using plugin io.cucumber.core.plugin.SerenityReporterParallel
the tests started to run twice(that wasn't the case in serenity3)

there's seems to be no issue with the glue or the features values.

Could you advise me which of these annotations to use with serenity4 🙏
either this

import net.serenitybdd.cucumber.CucumberWithSerenity;
@RunWith(CucumberWithSerenity.class)

or this one

import io.cucumber.junit.Cucumber;
@RunWith(Cucumber.class)

[wakaleo]

If you are running in JUnit 4 you need to use CucumberWithSerenity with the JUnit 5 legacy mode.