New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
compile-time disabled test not correctly handled by junit log #206
Comments
Hi, This is by design that all test cases, even skipped ones, appear in the final JUNIT. The tools I am using on top of boost.test for instance show me the number of skipped tests, and this is an important piece of information that needs to be reported to developers. This is also in line with other unit testing frameworks I am using in eg. Python, Java, Typescript, etc. What I would rather do is to show skipped tests in HRF and XML, and improve the logs of the JUNIT to indicate that the skipped test has been disabled at compile-time. Would you agree with this resolution? PS: please avoid things like |
Hi, Thanks for looking into this. It's been a while and I don't fully recall all the details. P.S. sorry for the "it sucks" left in the code sample. Probably was having a though debugging session. This was meant at my computer not at boost.test. |
Thanks for the feedback (and the edit!). I understand now better the issue and may have a fix for this on branch |
Still some corner cases to test, but I believe the issue is fixed. |
Thanks a lot for the fix. I had a quick look at the code and it looks promising to me. Unfortunately I don't have the time to test it in detail, especially because I don't work at the company where we ran into this issue anymore. But I forwarded the thread to one of my former colleagues, maybe he'll have time to verify it. |
Thanks for the fix, as well! We have multiple test dashboards internally where we can see the discrepancy (testsuite attribute values vs number of skipped tags). We'll be able to verify this fix once there is a rc/beta build of boost 1.72. (patching etc is currently out of scope unfortunately). |
…ernal * topic/GH-206-compile-time-disabled-junit: Change log Fix counting the skipped tests Reverting unrelated breaking change
There is enough coverage for now. There might be additional issues because of the test tree traversal (breath first, not exploring disabled test units). In case you hit such an issue, please open a new ticket. |
In master, closing. |
Test cases can be disabled at compile-time or at runtime as explained in the doc here: https://www.boost.org/doc/libs/1_66_0/libs/test/doc/html/boost_test/tests_organization/enabling.html
In general, compile-time disabled tests do not appear in the log, and runtime disabled tests do. At least that is the case for HRF and XML format.
The JUnit log formatter does not follow this. As a result compile-time disabled tests appear in the JUnit report and in addition the "skipped" attribute of the testsuite is wrong and do not account for these.
Here is a sample code to reproduce the problem:
The text was updated successfully, but these errors were encountered: