New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v3 Report logs pass as skip in retry #680

Open
saikrishna321 opened this Issue Nov 9, 2016 · 12 comments

Comments

Projects
None yet
4 participants
@saikrishna321

@anshooarora

i have retry on failure, so when my test fails the first time(Skip) and passed the next run ( Pass), report should consider it as Pass and not as Skipped

Refer the screenshot below
screen shot 2016-11-09 at 6 23 10 pm

@saikrishna321

This comment has been minimized.

Show comment
Hide comment
@saikrishna321

saikrishna321 Nov 10, 2016

@anshooarora ping, this is giving a wrong test results report.. can you pls help

@anshooarora ping, this is giving a wrong test results report.. can you pls help

@anshooarora

This comment has been minimized.

Show comment
Hide comment
@anshooarora

anshooarora Nov 10, 2016

Owner

This is because SKIP has higher priority than PASS. There was an old discussion due to which I had set the priority like so, otherwise PASS always had higher priority.

Owner

anshooarora commented Nov 10, 2016

This is because SKIP has higher priority than PASS. There was an old discussion due to which I had set the priority like so, otherwise PASS always had higher priority.

@saikrishna321

This comment has been minimized.

Show comment
Hide comment
@saikrishna321

saikrishna321 Nov 10, 2016

@anshooarora this is giving a wrong feel, the test case skipped the first time and passed the second time on re-run .

Looking at skip, it gives a wrong data on the dashboard and we panic seeing the reports on pipeline and when your on CI/CD which is very aggressive

Do you think we need to consider FATAL, FAIL, ERROR, WARNING, PASS, SKIP, INFO, WARNING ?

saikrishna321 commented Nov 10, 2016

@anshooarora this is giving a wrong feel, the test case skipped the first time and passed the second time on re-run .

Looking at skip, it gives a wrong data on the dashboard and we panic seeing the reports on pipeline and when your on CI/CD which is very aggressive

Do you think we need to consider FATAL, FAIL, ERROR, WARNING, PASS, SKIP, INFO, WARNING ?

@anshooarora

This comment has been minimized.

Show comment
Hide comment
@anshooarora

anshooarora Nov 10, 2016

Owner

I will let this hierarchy remain but provide a way to setup your own hierarchy. Would that work?

Owner

anshooarora commented Nov 10, 2016

I will let this hierarchy remain but provide a way to setup your own hierarchy. Would that work?

@saikrishna321

This comment has been minimized.

Show comment
Hide comment
@saikrishna321

saikrishna321 Nov 11, 2016

cheers , that works :-)

cheers , that works :-)

@SrinivasanTarget

This comment has been minimized.

Show comment
Hide comment
@SrinivasanTarget

SrinivasanTarget Nov 11, 2016

@anshooarora Can we have this in 3.0.1?

@anshooarora Can we have this in 3.0.1?

@anshooarora

This comment has been minimized.

Show comment
Hide comment
@saikrishna321

This comment has been minimized.

Show comment
Hide comment
@saikrishna321

saikrishna321 Nov 13, 2016

@anshooarora any tests for this ? which would help on how to use it ..

@anshooarora any tests for this ? which would help on how to use it ..

@anshooarora

This comment has been minimized.

Show comment
Hide comment
@anshooarora

anshooarora Nov 13, 2016

Owner

@saikrishna321 - docs & tests coming soon..

quick usage:

List statusHierarchy = Arrays.asList(
                    Status.FATAL,
                    Status.FAIL,
                    Status.ERROR,
                    Status.WARNING,
                    Status.SKIP,
                    Status.PASS,
                    Status.DEBUG,
                    Status.INFO
);

extent.config().statusConfigurator().setStatusHierarchy(statusHierarchy);
Owner

anshooarora commented Nov 13, 2016

@saikrishna321 - docs & tests coming soon..

quick usage:

List statusHierarchy = Arrays.asList(
                    Status.FATAL,
                    Status.FAIL,
                    Status.ERROR,
                    Status.WARNING,
                    Status.SKIP,
                    Status.PASS,
                    Status.DEBUG,
                    Status.INFO
);

extent.config().statusConfigurator().setStatusHierarchy(statusHierarchy);
@saikrishna321

This comment has been minimized.

Show comment
Hide comment
@saikrishna321

saikrishna321 Nov 13, 2016

can this be used in current version or next release ?

can this be used in current version or next release ?

@anshooarora

This comment has been minimized.

Show comment
Hide comment
@anshooarora

anshooarora Nov 13, 2016

Owner

3.0.1 onwards.

Owner

anshooarora commented Nov 13, 2016

3.0.1 onwards.

anshooarora added a commit that referenced this issue Nov 14, 2016

@fragonib

This comment has been minimized.

Show comment
Hide comment
@fragonib

fragonib Nov 9, 2017

Customizable status hierarchy is great. Thank you.

What do you think about letting the user to express status agregation algorithm for any group level (ie.: test groups and suites). It would be more flexible and customizable than simply using status hierarchy. Is this possible currently? Otherwise I don't know the impact of such feature on current code but please, consider it.

fragonib commented Nov 9, 2017

Customizable status hierarchy is great. Thank you.

What do you think about letting the user to express status agregation algorithm for any group level (ie.: test groups and suites). It would be more flexible and customizable than simply using status hierarchy. Is this possible currently? Otherwise I don't know the impact of such feature on current code but please, consider it.

@anshooarora anshooarora reopened this Nov 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment