JUnit scheme compatibility and Jenkins xUnit plugin 2.0.x

[pthorod]

Hi

I have been successfully running Jenkins with xunit plugin to process test result reports from CppUTest up until upgrading Jenkins xUnit plugin to 2.0.x. After the upgrade, xUnit plugin states that the xml output from CppUTest is invalid for the metric "JUnit"

The 2.0.0 changelog states stricter valdiation of the xml format and breaking changes/support. My question is what "JUnit" scheme CppUTest actually outputs, and if I should simply use another configuration, or if something should be updated on the CppUTest side.
xUnit plugin allows also to select "Custom tool" and supply own XLS file.

quote from Jenkins xUnit plugin changelog for 2.0.0 :

I have introduce some schemas validation becase the large number of issues opened to xUnit plugin. Those issues are caused by custom attributes or XML DOM in the report generated by some user/tools report logger also if they declare to produce a valid report file. This cause me to spend a lot of time to investigate what is right and what is wrong and how to implement the XSL without it goes in error during transformation.

Official supported JUnit schema are Ant junit and Maven Surefire).
In case you are using a non java tools that declares to produce a JUnit compatible report, ensure the report respects one of the widely used supported schemas. If not than select as test type the build step "Custom Tool" and provide a XSL as described here.

NUnit 2.x (2.0 ~ 2.6) schema validation from official distribution.
In case you are using a tools that declares to produce a NUnit 2 compatible report, ensure the report respects the official schemas. If not than select as test type the build step "Custom Tool" and provide a XSL as described here

I upgraded to newest CppUTest and build from source, revision 2edc6bb
Caling test executable with -v -ojunit arguments.

Jenkins xUnit plugin configuration: "JUnit"

Jenkins console:

22:06:02 INFO: Processing JUnit
22:06:02 INFO: [JUnit] - 1 test report file(s) were found with the pattern 'build/common/simplelib_test/*.xml' relative to 'C:\jenkins\workspace\cmakeExample' for the testing framework 'JUnit'.
22:06:02 WARNING: The file 'C:\jenkins\workspace\cmakeExample\build\common\simplelib_test\cpputest_add_test.xml' is an invalid file.
22:06:02 WARNING: At line 5 of file:/C:/jenkins/workspace/cmakeExample/build/common/simplelib_test/cpputest_add_test.xml:cvc-complex-type.3.2.2: Attribute 'assertions' is not allowed to appear in element 'testcase'.
22:06:02 WARNING: At line 5 of file:/C:/jenkins/workspace/cmakeExample/build/common/simplelib_test/cpputest_add_test.xml:cvc-complex-type.3.2.2: Attribute 'file' is not allowed to appear in element 'testcase'.
22:06:02 WARNING: At line 5 of file:/C:/jenkins/workspace/cmakeExample/build/common/simplelib_test/cpputest_add_test.xml:cvc-complex-type.3.2.2: Attribute 'line' is not allowed to appear in element 'testcase'.
22:06:02 FATAL: The result file 'C:\jenkins\workspace\cmakeExample\build\common\simplelib_test\cpputest_add_test.xml' for the metric 'JUnit' is not valid. The result file has been skipped.
22:06:02 org.jenkinsci.plugins.xunit.service.TransformerException: The result file 'C:\jenkins\workspace\cmakeExample\build\common\simplelib_test\cpputest_add_test.xml' for the metric 'JUnit' is not valid. The result file has been skipped.

Output test report file:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="0" failures="0" hostname="localhost" name="add_test" tests="1" time="0.000" timestamp="2018-06-29T22:06:02">
<properties>
</properties>
<testcase classname="add_test" name="add_returns_correct_result" assertions="1" time="0.000" file="C:\jenkins\workspace\cmakeExample\src\common\simplelib_test\add_test.cpp" line="18">
</testcase>
<system-out></system-out>
<system-err></system-err>
</testsuite>

Thanks for a great tool, let me know if you need more information or if I can contribute in some way.

[pthorod]

There are of course quite some parsing bug reports on the xUnit plugin issue tracker after 2.0.x. But would like to at least verify my settings and understand what CppUTest is supposed to support before filing a bug report there.

https://issues.jenkins-ci.org/browse/JENKINS-52253?jql=project%20%3D%20JENKINS%20AND%20component%20%3D%20xunit-plugin%20ORDER%20BY%20created%20DESC

[basvodde]

Thanks. I won't look at it right now but some time later. Strange though.

[jgonzalezdr]

Unfortunately there is no such thing as an official JUnit output xml schema, the xUnit plugin devs just reverse engineered it from JUnit source code, so the left it to the minimum without any extension capability in the schema that they use.

The easiest option is to use an XSL file that removes "extra" attributes and leaves a bare JUnit output and load it to the Jenkins plugin as described in the plugin changelog.

A better option would be to implement or convince xUnit plugin devs to implement an embedded tool for CppUTest output.

[raffienficiaud]

I found this useful for junit:
http://llg.cubic.org/docs/junit/