Serenity summary full report does not show correct count of test cases

[akshayrane308]

When a scenario is "Broken" (test result status: Broken) in the @before code i.e. before executing the actual scenario steps, the full report does not show the correct count of test cases. The report shows the total tests on the very first page (dashboard) as the count of all scenario steps instead of the count of the scenarios/scenario outlines. Also, the Test outcomes and performance graphs too shows the incorrect count

Dummy example:

Consider below scenario with 9 steps:

Given I am on the serenity shopping site
And I search for iPhone X
When I add iPhone X to the cart
And proceed to checkout
And enter billing address
And enter shipping address
And enter payment details
And place the order
Then I should see the order placed confirmation message

Assume the scenario has below Serenity steps (annotated by @step) in Before Hook annotation:

1. Verify homepage is loaded completely
2. Dismiss all advertisement notifications
3. Click on Login
4. Enter username
5. Enter password
6. Click on Login
7. Verify dashboard shows user icon with name

Assume the scenario has below Serenity steps (annotated by @step) in After Hook annotation:

1. Click on user icon in page header
2. Click on logout button
3. Verify logout successful popup is displayed
4. Verify dashboard page shows link to login in the page header

Now, assume the scenario runs in below manner:

1. Scenario ends in status error i.e. broken at the @before hook step "Verify dashboard shows user icon with name" due to page did not load due to sudden server down
2. The actual scenario tests are skipped
3. The @after hook steps are executed successfully

The serenity full report shows the test count as below:

Total Tests: 14 (1 + 9 + 4)
Pass: considers all 4 @after hook steps as individual test case
Skipped: considers all 9 Scenario steps as individual test case
Fail: considers entire @before hook steps as single test case

Expectation:

Total Tests: 1
Pass: 0
Broken: 1
Fail: 0

@wakaleo I have been struggling with this since when I started using serenity from last 2 years. Can you please check this and let me know if this can be fixed or otherwise.

[akshayrane308]

Facing this in latest serenity-maven-plugin v3.9.8 (since v2.0.91 when I started using serenity)

[wakaleo]

Can you provide a sample project to reproduce the issue?

[wakaleo]

I can't reproduce this issue.

[akshayrane308]

Working on creating the sample project to reproduce the issue. I will keep you updated.

[Darenth]

Thank you @akshayrane308 for trying to create the project for reproduction. I have the same problem, but cannot reproduce it outside of my work. The problem exists for me too, and I don't how to find out where is the problem.

[wakaleo]

Make sure you are using serenity-core 3.9.8 or higher

[Darenth]

@wakaleo thanks for your quick response, but I'd like to use 3.9.8 instead of 3.9.7, but I cannot because I don't know where Injectors.java from guice (serenity model) was moved. Could you please provide how to import Injectors? or they were deprecated?

[wakaleo]

Guice is no longer supported - if you are calling the Serenity internals, you should now use either the SerenityInfrastructure class, the ModelInfrastructure class, or (for EnvironmentVariables), just declare the field.

[Darenth]

Today I tested version 4.0.0.beta-1 and had the same situation with wrong count tests. Serenity treats every step in the test as a test case.

[wakaleo]

I have never seen this behaviour. Can you provide a sample project that reproduces this issue?

[Darenth]

I cannot do that because it is hard to reproduce, I hope that @akshayrane308 can create the sample project with this issue.

[wakaleo]

The problem may be project-specific. In that case, a commercial support contract would be the best way to resolve the issue quickly.

[akshayrane308]

I am working on this, will keep you posted

[Darenth]

@akshayrane308 , Thanks! I have been waiting for your work

[Darenth]

@akshayrane308 Hello, do you have any updates from Your side?

[wakaleo]

Try with 4.0.12.

[Darenth]

Try with 4.0.12.

Tried, the same situation.
@akshayrane308 Hello, do you have any updates from Your side?

[wakaleo]

@Darenth Without a project to reproduce this issue, it will not be possible to fix outside of a commercial support package (where we can look at your code directly).