Scenario count (including rows of test data) is not updated properly in Serenity 2.0.xx

[hariharanravichandran]

Serenity Core Version: 1.8.1
Serenity Cucumber Version: 1.6.5

Serenity Core Version: 2.0.56
Serenity Cucumber Version: 1.9.40

Scenario Results (including rows of test data) should be Total of 168 in which 166 Automated and 2 Manual. But, it is displayed as 93 Total in which 91 Automated and 2 Manual.

Similarly for feature file having 2 Scenario outline(totals 6 scenarios), the count is mentioned as 2 in the Functional Coverage Overview instead of 6.

[wakaleo]

Can't reproduce with serenity-core 1.0.70 and serenity-cucumber 1.0.25. Can you provide a sample project that reproduces the issue?

[hariharanravichandran]

Hi @wakaleo , Issue is resolved in the version mentioned(serenity-core 1.0.70). But, the issue is still reproducible when we execute our tests in Jenkins.

[hariharanravichandran]

This issue happens in Zalenium - Jenkins when we execute multiple features with scenario outline. When we execute a single feature with Scenario outline, Report is generating correctly

[hariharanravichandran]

Hi @wakaleo , I am not able to run the test using -Dcucumber.options.

[wakaleo]

Do the multiple files have different feature and scenario outline names?

[hariharanravichandran]

Yes. multiple files have different features and scenario outline names. There is one more weird observation. When I executed my whole test suite in Zalenium(Jenkins) without the -Dcucumber.option"--tags @something", the total scenario count is displayed as expected.

[RAS-2112]

I've been able to re-create this issue in serenity-cucumber4-starter by using Dfailsafe.rerunFailingTestsCount. When re-running failures in a scenario outline, the successful tests are still counted in the overall test count, but not in scenario results.

Full config:

clean verify "-Dcucumber.options=--tags '(@Test)'" -Dfailsafe.rerunFailingTestsCount=1

Running the above Maven command in this sample project should show you the issue.

[wakaleo]

Thanks for reproducing this. There isn't any special support for the failsafe retries feature, so apparently something about the. way it records the results confuses the Serenity reporting. If you want to have a look at the code to see what is going on, I'd be happy to review a PR.