Retry report showing wrong scenario status if finally the step has passed

[authorjapps]

From Slack:
Cyril EDME:
Hi,
i would like some feedback on a use case before post any ticket.
Did someone created a scenario with a first step that trigger an asynchrone treatment then assert it with a second step (sync, by REST API using the retry policy) ?
In the end it works, my second step first attempt fail but the second succeed, perfect. But in the report it's recorded as fail (even if the test is green), which is incorrect. The detail of the steps with the first attempt in fail status is correct tho.

---

But in the report it's recorded as fail (even if the test is green), which is incorrect.
@ Cyril EDME, the issue you are facing is the report issue when retry mechanism is used?
e.g.

"retry":{
   "max": 5
   "delay": 500
}

Even if let's say first 4 retries failed and 5th one passed, the test goes green(expected), but report shows FAILED (unexpected)?

Is the report missing the 5th entry which has PASSED ?

Can you paste here a result of just one test-scenario run's
Console log : with all retry entries
CSV report : what it has captured
HTML file

Just for 1 test-scenario please which will be simple to analyze and less noisy!

---

The reporting code is below which marks the Scenario as failed if it has at least 1 step failure.

org.jsmart.zerocode.core.report.ZeroCodeReportGeneratorImpl#generateExtentReport

This behaviour is correct in a normal usecase, but not for a retry-step usecase.

Possible fix:

• Check if one of the retries has passed, then only mark the scenario as PASSED.
  • Basically if one of the retries succeeds, then it doesn't retry any more. e.g. if max retry is 5, and the 3rd once succeeds, then it comes out from that step(does not retry for 4th and 5th time) and continues for the next step.
• Then see if the Extent report is correctly rendering this i.e. marking the Scenario as PASSED

Download and unzip this - retry html and csv report.zip

AC1:

Mark the scenario as Pass(Green) if any of the retries succeeds for the retry-step(s) and all other steps in this scenario have passed.

AC2:

Remove the redundant failed retry steps from extent-report view. This will leave with the final outcome of the step(wither Passed or Failed) with just the 1 entry which is the correct for the HTML report view.
In case the test-engineers need to see the failed retry steps, then they can look at the logs in the target folder.

AC3:

If AC2 solution is implemented, then remove the redundant failed retry steps from CSV records to make both consistent with each other.

AC4(Tech-Debt):

Create a tech-debt ticket to raise an issue with Extent Project to allow overriding the final scenario status(green or Pass in this case) even if the scenario has a failed step. This seems like not possible currently.
First check in the Extent Slack channel prior to raise as issue.

[sparrowV]

@authorjapps The scenario should be marked as PASSED if one of the retries succeed , not just last one right?

Ans: PASSED only when any retry succeeds. Basically if one of the retries succeeds, then it doesn't retry any more. e.g. if max retry is 5, and the 3rd once succeeds, then it comes out from that step(does not retry for 4th and 5th time) and continues for the next step.

In the report scenario marked as PASSED do you mean the marking on the picture bellow?

Ans: Yes. You are right. That encircled scenario should be marked as Pass (Green)

[authorjapps]

@sparrowV , I have answered them inline, hope it is clear.

[sparrowV]

@authorjapps yeah, I think its clear. I will assign it to myself, if you are ok with it

[authorjapps]

Available in V 1.3.20