Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

results displayed in the Run Summary & Slack report are inconsistent, due to counting retries #384

Closed
alexniculae opened this issue Jun 17, 2021 · 0 comments · Fixed by #427
Assignees
Labels
bug Something isn't working

Comments

@alexniculae
Copy link

Summary

The results displayed in the Run Summary are inconsistent, probably due to the program counting the number of retires in the Total Tests count.

This also seems to affect the Slack integration report, where a failed test is counted each time a retry is executed, and the total number of tests will be artificially increased.

In the below example image, we can see that the number of passed and failed tests is correct, but the total number of tests is miss-calculated.
image

The issue is propagated a bit differently in the Slack report, where the number of failed tests is higher, as it considers all the failed retries of each test as well, instead of counting just the end result.
image

How to reproduce

Force-fail some tests in a run recorded in sorry-cypress, with retries turned on; this should be the only required element.

@agoldis agoldis self-assigned this Jun 17, 2021
@agoldis agoldis added the bug Something isn't working label Jun 17, 2021
@agoldis agoldis mentioned this issue Aug 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants