Failure not being reported up when last keyword passes

[skinny-b]

Was running tests that have an intermittent failure and wasn't seeing the failure being reported to the top level. It shows reduced testcase count, as test didn't complete, but no failures. With the help of @iivanou (thank you) were able to determine that the ending keyword pass seems to be the culprit.

Using the following:
(venv) bash-4.2$ pip list | grep report
pytest-reportportal 1.0.8
reportportal-client 3.2.3
robotframework-reportportal 3.1.0

Last image shows the red ones as failed and the last Capture as pass.

[iivanou]

@skinny-b it can be fixed on the agent side. But, firstly, I'd like to ask @DzmitryHumianiuk about limitations from the application side. Is that kind of expected behavior when TestSuite takes into account the result of the last nested element only?

[DzmitryHumianiuk]

If you populate status in FinishRQ RP set it to the item.
If status is not provided, than it calculated based on childs.

• if FAILED>0 then set FAILED

if you finish parent (suite) level before child (test), then child will receive status of parent, and in case of finishRQ will have status for child, it will be updates afterwards.

hope this helps,
or i didn't get the question

[iivanou]

The problem is that RP shows up the wrong summary for the test. In Robot Framework test case consists of keywords that are being reported as children to the high-level keywords in a nested manner. On the screenshot below we can notice that test case reports 5 passed steps(children keywords) instead of one failed(high-level keyword). I think it's not correct. If we disable reporting for nested keywords in the agent, the results are reported as expected.

[DzmitryHumianiuk]

@iivanou in version 5 we have reportportal/reportportal#275 for this.
so you can just add steps into the test object.

we can have a call next week with dev team, to discuss the way it works

[iivanou]

@DzmitryHumianiuk oke, let's do this. Getting back to the issue. Our problem is hidden in line 204. We consider the last element of the self.stack being the parent_id for the current item that makes us unable to control nested structures. We can't modify this line without breaking other Python agents.
The only thing we can do here is to make changes in the start_test_item interface of the client in order to provide agents with an ability to control parent_item_id value. Fix for this issue requires changes both in common client and RF agent. I do not see an easy way to work it out.

[iivanou]

@skinny-b could you please run the same test suite against the v5.0 of RP? https://beta.demo.reportportal.io/

[DzmitryHumianiuk]

@iivanou does async capabilities in RPv5 will help to solve it?

now you can create tree structure of elements on client side, by generating item UUID on client side, and send it in HTTP request to server.
So client side knows where item should belong to, and server can accept create requests in any sequence. Even child can be sent before parent.

[iivanou]

@DzmitryHumianiuk it should, definitely. But it will require RP to be upgraded to V5.

[DzmitryHumianiuk]

@iivanou

it should, definitely. But it will require RP to be upgraded to V5.

sure,
i guess we need to keep it as separate branch for v5

[skinny-b]

I don't believe I can test it on V5 yet, once reportportal/reportportal#715 is resolved we can deploy it to test it further.

[iivanou]

@skinny-b It's already deployed for everyone: https://beta.demo.reportportal.io/. You can use your GitHub account to log in.

[skinny-b]

Saw the same against the beta.demo.reportportal.io server: Testcase data is altered slightly but resulting on failure take a screenshot which ends up passing seems to show the failure.

[iivanou]

@skinny-b Unfortunately, we have to change a common client and agents to make use of new async reporting introduce in RP v5. I'll let u know when a list of necessary changes is ready to go.

[iivanou]

@skinny-b Can u give a try for the changes in PR #40? Let us know the results. Thanks.

[skinny-b]

Tried with both #40 and reportportal/client-Python#62 and got the same result as before: #39 (comment)

[iivanou]

Ok, in this case, I would consider reportportal/reportportal#275 feature is not working by design.

Status of test item with nested steps:
If all nested steps are passed - Parent item is passed
If one of the nested steps is failed - Parent item is failed

In our case, we have at least one failed children, but RP still reports all passed. @DzmitryHumianiuk, test items with type "STEP" are still being reported with method type "TEST". What type should be set for nested steps? I couldn't find this information at https://beta.demo.reportportal.io/ui/#api.

[evjlobanova]

@Yumfriez Vanya, could you please help? Could you please add a link to the dev guide for Nested steps

[evjlobanova]

@iivanou it should be STEP

Nested step marker is parameter:
"hasStats": FALSE

{
 "attributes": [
 {
 "key": "string",
 "system": true,
 "value": "string"
 }
 ],
 "description": "string",
"hasStats": FALSE,
 "launchId": "string",
 "location": "string",
 "name": "string",
 "parameters": [
 {
 "key": "string",
 "value": "string"
 }
 ],
 "retry": true,
 "startTime": "2019-06-21T14:50:15.600Z",
 "type": "SUITE",
 "uniqueId": "string"
}

[evjlobanova]

Dev guide for Nested steps:
https://github.com/reportportal/client-java/wiki/Nested-steps
https://github.com/reportportal/client-java/wiki/Step-template-and-step-template-config

[Dhams-1]

@iivanou Hi is there any update on this issue. kindly let me know when are we planning to fix this issue or how can I over come this. Thanks for your time

[iivanou]

This issue will be fixed in new agents that will support RP version 5 only.

[Dhams-1]

@iivanou Thank you.