-
-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[💡 Feature]: Wdio spec reporter should exclude hook failures in test case pass fail count #12524
Comments
Thanks for raising this issue 🙏 Unfortunately we can't help you without a reproducible example in this matter. Please read our contributing guidelines on how to create a reproducible example. If you can't provide a reproducible example we will close this issue in 7 days. If you have any questions please reach out to us on our Discord channel. We are happy to help you out there. |
I did a quick check and what we do is we put the failed hook in the list of failed specs. It would require a rewrite of this logic, any help on this is welcome and appreciated. |
Thanks for reporting! We greatly appreciate any contributions that help resolve the bug. While we understand that active contributors have their own priorities, we kindly request your assistance if you rely on this bug being fixed. We encourage you to take a look at our contribution guidelines or join our friendly Discord development server, where you can ask any questions you may have. Thank you for your support, and cheers! |
@erwinheitzman can you provide a link to the locations you looked into? |
The above reporter shows a spec report 🤔 do we see the same behavior across multiple reporters here? |
Is your feature request related to a problem?
Yes, I am using "@wdio/junit-reporter": "^8.32.4", and getting report with beforeAll Hook and afterAll Hook failed from the spec-reporter report like below. In case of success, it excludes the hooks in spec reporter. So I get 4 passed. In case of failure, it includes the hooks in spec reporter. So I get 6 failed. In success/failure, the total count should not be changed for me. This is shown in Dashboard in graphs making difficult to understand by business team, why it shows count less one day (in case of success) and other day count more (in case of failure).
[edge-headless-shell 122.0.2365.52 windows #0-0] Spot - A and P
[edge-headless-shell 122.0.2365.52 windows #0-0] Verify whether the user is able to search with a ID in Spot screen
[edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate A and P tab is Highlighted
[edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate RMC Menu Details
[edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate Export RMC Menu
[edge-headless-shell 122.0.2365.52 windows #0-0] ✖ "after all" hook
[edge-headless-shell 122.0.2365.52 windows #0-0]
[edge-headless-shell 122.0.2365.52 windows #0-0] Verify No records text in A and P Tab
[edge-headless-shell 122.0.2365.52 windows #0-0] ✖ "before all" hook
[edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate No Records
[edge-headless-shell 122.0.2365.52 windows #0-0]
[edge-headless-shell 122.0.2365.52 windows #0-0] 6 failing (27.9s)
Describe the solution you'd like.
Exclude the hooks count in total test cases passed or failed. Hooks are meant to setup initial or final state of the application/test suite.
Reference to Discord channel:
https://discord.com/channels/1097401827202445382/1097674067601010709/1218898644284080208
Describe alternatives you've considered.
Not sure, as Junit or HTML reports generated on the basis of default spec reporter generated, in the spec reporter itself this issue is seen.
Additional context
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: