Tests in Scenario Outlines tagged with "@manual-result:success" are ignored from the report indicators

[mathfalk]

Hi, I'm currently using Serenity v3.4.2 with Cucumber in a project that has 2 features, containing 5 scenario outlines (totalizing 12 rows of test) and 18 common scenarios.

All of them are tagged with "@Manual" and "@manual-result:success", but only the common scenarios are counted as Passing tests in the pie chart, while the others are completely absent from it.

This causes inconsistencies not only in the pie chart percentages, but also in the Test Outcomes chart, functional coverage overview and functional coverage details sections, as shown in the screenshots below.

[wakaleo]

Please try with 3.6.12 (and see https://tealfeed.com/include-manual-tests-serenity-reports-roytk)

[mathfalk]

Hi John, thanks for answering.

I updated to 3.6.12 as you suggested and the 12 tests previously missing from the charts are now being counted as 'Ignored Test Cases', which is also incorrect since these scenarios are all tagged with manual-result:success (or passed).

I also checked the article you provided, but it doesn't address this specific case (Scenario Outlines with multiple rows of test data).

[wakaleo]

Are you running the features with the JUnit 5 runner?

[mathfalk]

I'm running with the CucumberWithSerenity runner (couldn't get Serenity to work with Cucumber and JUnit5). I used serenity-cucumber-starter as the template for this project's pom.xml and suite class.

[wakaleo]

Try with the JUnit 5 runner - the fix has only been made there:

import org.junit.platform.suite.api.IncludeEngines;
import org.junit.platform.suite.api.SelectClasspathResource;
import org.junit.platform.suite.api.Suite;

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("/features")
public class AcceptanceTestSuite {}

And the Cucumber JUnit 5 dependencies:

        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-junit-platform-engine</artifactId>
            <version>7.2.3</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-suite</artifactId>
            <version>1.8.2</version>
            <scope>test</scope>
        </dependency>

[mathfalk]

Changing to JUnit 5 caused the aggregate report to be generated with 0 tests. Also tried these changes in a fresh clone of the serenity-cucumber-starter project, but the result was the same.

[wakaleo]

I will write some docs on the JUnit 5 integration in the coming days. Not sure if/when this fix will be back-ported to the JUnit 4 runner.

[mathfalk]

I managed to fix the JUnit 5 integration in my project by adding the junit-platform.properties, as explained here: https://serenity-bdd.github.io/docs/guide/cucumber.
Running the tests with JUnit 5 didn't solve my original problem though, and the tests from scenario outlines are still being computed as Ignored in the report.

Edit: I attached a simple project to demonstrate this bug.
serenity-cucumber-starter.zip

[szead]

Hey @mathfalk,

I'm just playing around with the @manual reporting functionality of Serenity and I have the same issue: cucumber marks everything ignored, even if it is manual. I recently tried to add the @wip tag as well, and it seems they are passing now.
I'm not sure this is the right solution, however, but at least it is working :)

[mathfalk]

Hey @szead, thanks for the tip, but I'm not sure my issue is the same as yours, as in my case, only scenario outlines were affected.

Anyway, I checked again after updating Serenity to 3.6.22 and confirmed that this issue has been fixed, with all test results being correctly counted in the report (at least using the JUnit 5 runner).