Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

junit report: test error is also reported as failure #196

Closed
mistafunk opened this Issue Jan 21, 2019 · 5 comments

Comments

Projects
None yet
2 participants
@mistafunk
Copy link

mistafunk commented Jan 21, 2019

Consider this snippet:

#define BOOST_TEST_MODULE Test1
#include <boost/test/included/unit_test.hpp>

BOOST_AUTO_TEST_CASE(testexception)
{
 throw 42;
}

Compiling and running above sample yields:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite tests="0" skipped="0" errors="1" failures="1" id="0" name="Test1" time="0.000137">
<testcase assertions="1" name="testexception" time="0.000137">

Now, this is not so great. JUnit would report this as:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite tests="0" skipped="0" errors="1" failures="0" id="0" name="Test1" time="0.000137">
<testcase assertions="1" name="testexception" time="0.000137">

Because according to JUnit a test either succeeds or fails (one or more assertion failures) or produces an error (unexpected failure). Boost on the other hand thinks that an error is also a failure.

Reproduced with Boost 1.66.0

@raffienficiaud raffienficiaud self-assigned this Feb 8, 2019

@raffienficiaud

This comment has been minimized.

Copy link
Member

raffienficiaud commented Feb 8, 2019

If I understand well, as soon as there is an error, there is no failure? Can it happen that there are several errors?

@raffienficiaud

This comment has been minimized.

Copy link
Member

raffienficiaud commented Feb 11, 2019

I think I fixed this in branch topic/GH-196-junit-abort-fail-not-same. It would be good if you have a look to tell me if I understood the difference between errors and failures in the JUnit sense.

Thanks!

@mistafunk

This comment has been minimized.

Copy link
Author

mistafunk commented Feb 15, 2019

@raffienficiaud i understand junit report semantics like this:

  • failure: a test case with one or more failed assert-like statements (this is a bit contested, there are other frameworks which count each assertion failure)
  • error: a runtime error in the test case (i.e. bad test case implementation)

to your questions:

  • technically, there could be first a failed assertion and then an unexpected/unhandled exception or segfault within a test case. junit report ignores such assertion failures.
  • while there could be multiple implementation issues in a test case, the first one is usually severe enough and that's the error.
@mistafunk

This comment has been minimized.

Copy link
Author

mistafunk commented Feb 15, 2019

@raffienficiaud you branch looks fine

raffienficiaud added a commit that referenced this issue Mar 1, 2019

Merge branch 'topic/GH-196-junit-abort-fail-not-same' into next-internal
* topic/GH-196-junit-abort-fail-not-same:
  Change log
  Junit counting fixups: failed and errored differences

@raffienficiaud raffienficiaud added next and removed fix-proposed labels Mar 1, 2019

@raffienficiaud raffienficiaud added this to the 1.70 milestone Mar 1, 2019

@raffienficiaud raffienficiaud added 1.70 and removed next labels Mar 1, 2019

@raffienficiaud

This comment has been minimized.

Copy link
Member

raffienficiaud commented Mar 1, 2019

In master, closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.