Bad scenario and step alignement in reports for Scenario Outline

[Rangiferus]

What happened?

Failed cases of scenarios defined as Scenario Outline are improperly aligned in reports.

1. Bad failed step treatment
   

2. Failed steps are reported as scenarios in the "Overall Test Results" tab.
   

3. The first failed test case is not listed in the scenario details view (when clicked on a failed scenario in the "Overall Test Results".
   

4. Failed and broken steps are improperly aligned in the expanded view.
   

What did you expect to happen?

No response

Serenity BDD version

4.0.14

JDK version

11

Execution environment

Maven, Chrome

How to reproduce the bug.

Create a Scenario Outline with Examples with failing and broken cases.

How can we make it happen?

Add it to the Serenity BDD backlog and wait for a volunteer to pick it up

[wakaleo]

Can you provide a sample project? This is very hard to reproduce and very tricky to fix, so I'm not sure when it will get looked at.

[Rangiferus]

I tried to reproduce this problem in a simplified project because I cannot upload the one I'm working on.
Unfortunately a similar scenario produces a decent report with only one problem showing up:

Here's the project: https://github.com/Rangiferus/basic_tg_serenity_project

[Rangiferus]

It's tough to reproduce these problems with messed up reports in a project separate from my work-related one but here's another example of failed and compromised cases from a scenario outline:

The table on top shows 3 cases: 1 failed and 2 compromised.
The steps listed below show only 2 cases (compromised ones). The expanded 3rd one shows strangely a broken step (java.lang.NullPointException). It's difficult to figure out what's going on here.

[wakaleo]

I tried to reproduce this problem in a simplified project because I cannot upload the one I'm working on. Unfortunately a similar scenario produces a decent report with only one problem showing up:

Here's the project: https://github.com/Rangiferus/basic_tg_serenity_project

In your scenario outline, each row in the example tables is a separate scenario and so will be reported as a separate test case (not as a step, although they appear as top-level steps in the test result page) - this seems normal in 4.0.18, so maybe I am not understanding the issue you are seeing?

Feature: [1] Scenarios
  As a tester that uses Serenity I want to reproduce problems from my original project

  @issue:#3833
  Scenario Outline: Broken and failed examples
    Some comment

    Given step to pass
    When step to <status>

    @tag1 @tag2
    Examples:
      | status |
      | fail   |
      | fail  |
      | break  |

    @tag2
    Examples:
      | status |
      | pass   |

    @known-issue
    @tag2
    Examples:
      | status |
      | pass   |

    @issue:#10374
    @tag1
    Examples:
      | status |
      | break  |

[Rangiferus]

Yest, that's true - 2 test cases were not executed in this example scenario.
Anyway, what the report says is inconsistent - there should be only 1 passing case (the other one is filtered out from the execution because it doesn't have @tag2 tag), 2 failed, 1 broken and 2 skipped (or ignored - whatever word is proper for unfiltered cases).
It's hard to understand the difference between "ignored" and "skipped" here.

As for the original problem which still messes up my reports, it must have something to do with the nested screeplay tasks mixed with other code that also calles other methods that include screenplay tasks.
So basically it's a result of using:

• scenario examples used to filter test cases to run based on tags
• step implementations that use not only screenplay tasks but also other code and that invokes shared methods with other screenplay tasks.

Maven runs print sensible output in the console but aggregated reports look totally bad and it's impossible to guess what happened with failed or broken cases.
The code is too complex to quickly extract as an example project for someone else to reproduce it.

[Johan-Sap]

I feel like the issue iam seeing at this point might be related to this one. 4.0.18 is the last version where I see it behaving as expected, after this version I see the following behaviour;

In case of a Scenario Outline, when an assertion (which is not in the StepsDefinitions but a layer deeper for instance in a Steps Libraray) fails or an exception is thrown, the report is not displaying the steps as expected.

First issue is the overview shows the tests as 'ignored' while they are in fact 'Failed'. That is not the worst because on the main overview page they are indicated as failed which is the page most people look at anyway.
However when looking into the steps the examples are actually merged into each other. The steps of the second example are becoming part of the failing step of the first example. This also results in the overview to show '1 skipped test' instead of 2 while when diving deeper there are in fact 2 examples. See screenshots

I have a demo project that consistently reproduces this issue. As said I only see the behaviour when the failing assert, or exception occurs on a deeper level than the StepDefinitions. We however, for instance, validate a response code to be 200 for API calls which are not always happening on that step definition level but deeper into the code. This will result in the above reporting behaviour.

In the demo project I run serenity version 4.0.29 and serenity-gradle-plugin 4.0.27. Downgraden these to 4.0.16 will at least fix the merging of the steps in case of a failure. It will however remain to show the steps as 'ignored' instead of 'failed'

Demo project : https://github.com/Johan-Sap/eu_pvh_demo2 (build with Gradle)

If this issue is in no way connected, or already know please let me know so I can either create a new issue for it or follow the already known issue.

[cliviu]

@Rangiferus , what was the latest version when you saw correct behaviour? We have checked the differences between 4.0.18 and 4.0.19 but we saw there no significant changes.

[Johan-Sap]

I have taken a closer look to the issue I am facing. Began to debug on the generation of the report and found out the issue is already present in the 'testOutcomes' object inside the ResultReportingTask (serenity-reports). Going back all usages I found out that the issue actually is already present inside the .json generated outcome file as you can see in below screenshots.

First screenshot is from 4.0.16 Serenity version where you can see the second 'example' is on the same level as the first one (child of the node 'testSteps'). Where in the second screenshot you see the generated file from the 4.0.19 Serenity version where that second testcase became a child from the first testcase.

Serenity 4.0.16 .json result file :

Serenity 4.0.19 .json result file :

Maybe this can help a bit in the search ?

[Johan-Sap]

@cliviu I searched a bit more and actually found a small code change between 4.0.18 and 4.0.19. Creating a snapshot with that change reverted solves my problems

In the snapshot I created I moved the currentStepDone method outside of the If statement. If I read the code correctly, when the currentStepDone method is INSIDE the if, it will only get triggered when the STEP Outcome is SUCCESS. (!hasStepFailed).

When debugging I see for a failed step it does not get into this if. That would ultimately result in not triggering the currentStepDone method. It would in my opinion make sense the testcase is not closed (step was never closed) and the first steps of the next example become part of the failed step of the previous example.

Below code works

I don't know why this code was moved into the IF statement so I am hesitant to make a pull request for this one. Maybe you (as you did the original change) still know what the reasoning behind it was and can make a better estimation on if reverting back will break other code.

I also realise now that my issue might not be as connected to the issue iam posting this in after all. If a new ticket is for this issue is required let me know. I will then create it.

[cliviu]

Hi, @Johan-Sap , the Pull Request is already done, it should be included in the next version. Thanks for your support

[Rangiferus]

Let's see if it helps.
@Johan-Sap I don't recall when I noticed my reporting problems for the first time - it must have been in or before October 2023 so definately version 4.0.18 was affected too. It even might be 4.0.12 or even 4.0.0.