Junit reporter with [!mayfail] tests

[jdumas]

Hi,

This is sort of a follow-up to #786. I have some unit tests marked as [!mayfail], so runnit unit tests with ctest return an error code of 0, even though those tests fail. However, when using the Junit reporter (which I need for Jenkins), those tests still appear as failed in the XML output. Would it be possible to mark them as skipped? It seems one can add a <skipped /> element under the test case like so:

    <testcase classname="test_bla.global" name="my_test" time="0.0" status="run">
      <skipped message="mayfail"/>
      <failure message="foo" type="REQUIRE">
FAILED:
...
      </failure>
    </testcase>

[horenmar]

Technically the change is easy enough, but with JUnit reporter the question always is what does the ecosystem at large do in similar cases. Can you point me to some resources/documentation on that?

[jdumas]

Hmm documentation on the XML schema is a bit hard to come by, but I think I saw this page and decided to try adding the <skipped> element to the XML output. I just tried it on my jenkins instance and it was enough to turn the "mayfail" test results from red (failed) to gray (skipped).

[horenmar]

Hmmm, I am not sure that marking run-but-failed-but-it-is-okay-to-fail tests as skipped is not too surprising. 🤔

[jdumas]

What other alternatives do we have? Currently if those tests fail, it is reported as an hard failure in the junit xml report. Thus making it impossible to distinguish between a "pass" and a "fail" based on the other test results. My understanding is that [!mayfail] is intended to be a temporary ductape for tests that can fail but may be fixed later, thus making it ok to mark them as "skipped" in the output. As far as I can tell junit doesn't have a notion of "this-test-failed-but-it's-ok", so really it's a best effort solution.

[horenmar]

Well, they can be reporter as passed. 😃

I'll have to think about this a bit.

[jdumas]

Sure, I'd be happy that they be reported as passed as well. Skipped seemed a good compromise to highlight failed tests that can be safely "ignored" for the purpose of the CI.

[jdumas]

@horenmar did you came to a decision regarding this issue?

[jdumas]

Hi. Wondering if there's any progress on this one?

[horenmar]

I am gonna go with "skipped", I am currently experimenting with different implementations.

[horenmar]

Ok, can you try whether the output from v2.x-junit-mayfail-as-skipped branch works for you?

[jdumas]

@horenmar sorry for the late reply. I tried the branch and can confirm that tests tagged as [!mayfail] now show up as skipped on my Jenkins. Thanks!