Skip to content
This repository has been archived by the owner. It is now read-only.

v3 Report logs pass as skip in retry #680

Open
saikrishna321 opened this issue Nov 9, 2016 · 12 comments
Open

v3 Report logs pass as skip in retry #680

saikrishna321 opened this issue Nov 9, 2016 · 12 comments

Comments

@saikrishna321
Copy link

@saikrishna321 saikrishna321 commented Nov 9, 2016

@anshooarora

i have retry on failure, so when my test fails the first time(Skip) and passed the next run ( Pass), report should consider it as Pass and not as Skipped

Refer the screenshot below
screen shot 2016-11-09 at 6 23 10 pm

@saikrishna321
Copy link
Author

@saikrishna321 saikrishna321 commented Nov 10, 2016

@anshooarora ping, this is giving a wrong test results report.. can you pls help

@anshooarora
Copy link
Owner

@anshooarora anshooarora commented Nov 10, 2016

This is because SKIP has higher priority than PASS. There was an old discussion due to which I had set the priority like so, otherwise PASS always had higher priority.

@saikrishna321
Copy link
Author

@saikrishna321 saikrishna321 commented Nov 10, 2016

@anshooarora this is giving a wrong feel, the test case skipped the first time and passed the second time on re-run .

Looking at skip, it gives a wrong data on the dashboard and we panic seeing the reports on pipeline and when your on CI/CD which is very aggressive

Do you think we need to consider FATAL, FAIL, ERROR, WARNING, PASS, SKIP, INFO, WARNING ?

@anshooarora
Copy link
Owner

@anshooarora anshooarora commented Nov 10, 2016

I will let this hierarchy remain but provide a way to setup your own hierarchy. Would that work?

@saikrishna321
Copy link
Author

@saikrishna321 saikrishna321 commented Nov 11, 2016

cheers , that works :-)

@SrinivasanTarget
Copy link

@SrinivasanTarget SrinivasanTarget commented Nov 11, 2016

@anshooarora Can we have this in 3.0.1?

@anshooarora
Copy link
Owner

@anshooarora anshooarora commented Nov 12, 2016

@saikrishna321
Copy link
Author

@saikrishna321 saikrishna321 commented Nov 13, 2016

@anshooarora any tests for this ? which would help on how to use it ..

@anshooarora
Copy link
Owner

@anshooarora anshooarora commented Nov 13, 2016

@saikrishna321 - docs & tests coming soon..

quick usage:

List statusHierarchy = Arrays.asList(
                    Status.FATAL,
                    Status.FAIL,
                    Status.ERROR,
                    Status.WARNING,
                    Status.SKIP,
                    Status.PASS,
                    Status.DEBUG,
                    Status.INFO
);

extent.config().statusConfigurator().setStatusHierarchy(statusHierarchy);
@saikrishna321
Copy link
Author

@saikrishna321 saikrishna321 commented Nov 13, 2016

can this be used in current version or next release ?

@anshooarora
Copy link
Owner

@anshooarora anshooarora commented Nov 13, 2016

3.0.1 onwards.

anshooarora added a commit that referenced this issue Nov 14, 2016
@fragonib
Copy link

@fragonib fragonib commented Nov 9, 2017

Customizable status hierarchy is great. Thank you.

What do you think about letting the user to express status agregation algorithm for any group level (ie.: test groups and suites). It would be more flexible and customizable than simply using status hierarchy. Is this possible currently? Otherwise I don't know the impact of such feature on current code but please, consider it.

@anshooarora anshooarora reopened this Nov 9, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.