Junit report is incompatible with Jenkins xUnit plugin #2221
Comments
|
We use the xUnit plugin instead of JUnit directly because we also have other non-JUnit tests (NUnit) which we want to have reported. |
|
How is this not filed as a bug against said Jenkins plugin instead? It's a valid number. This is ridiculous. |
|
I have reported the bug to Jenkins: https://issues.jenkins.io/browse/JENKINS-65438 |
|
by documentation the only supported JUnit schemas are: Ant junit and Maven Surefire |
horenmar
added a commit
that referenced
this issue
May 22, 2021
We used to use whatever precision we ended up having from C++'s stdlib. However, some relatively popular tools, like Jenkins, use Maven SureFire XML schema to validate JUnit test reports, and Maven SureFire schema requires the duration to have at most 3 decimal places. For compatibility, the JUnit reporter will now respect this limitation. Closes #2221
|
Well, I changed the JUnit reporter. However, I want to go on the record as saying that this is all extremely stupid, JUnit is bad and Maven surefire can go rot in hell. |
qtprojectorg
pushed a commit
to qt/qtbase
that referenced
this issue
Aug 5, 2021
The original Ant JUnit reporter produced test durations via Double.toString(), supporting arbitrary precisions, and the de-facto schema declared them as xs:decimal. Sadly, the now popular Maven Surefire reporter limited the duration to millisecond precision, and hard-coded this into its schema as SUREFIRE_TIME: https://issues.apache.org/jira/browse/SUREFIRE-1533 Unfortunately this definition spread into tools such as the Jenkins xUnit plugin, which relies on the schema provided by Maven Surefire: https://issues.jenkins.io/browse/JENKINS-52152 As a result, anything that produces higher precision results will not validate in the Jenkins xUnit plugin. Other test frameworks have bitten the bullet and reduced their precision correspondingly, e.g.: catchorg/Catch2#2221 catchorg/Catch2@581c46249acf8389e9 We follow suit, and our JUnit XML output now validates against both the Jenkins JUnit and xUnit plugins, as well as the original Apache Ant de-facto schema. Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: I3097d10c03c2a29709960372301b29055d224e10 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
qtprojectorg
pushed a commit
to qt/qtbase
that referenced
this issue
Aug 6, 2021
The original Ant JUnit reporter produced test durations via Double.toString(), supporting arbitrary precisions, and the de-facto schema declared them as xs:decimal. Sadly, the now popular Maven Surefire reporter limited the duration to millisecond precision, and hard-coded this into its schema as SUREFIRE_TIME: https://issues.apache.org/jira/browse/SUREFIRE-1533 Unfortunately this definition spread into tools such as the Jenkins xUnit plugin, which relies on the schema provided by Maven Surefire: https://issues.jenkins.io/browse/JENKINS-52152 As a result, anything that produces higher precision results will not validate in the Jenkins xUnit plugin. Other test frameworks have bitten the bullet and reduced their precision correspondingly, e.g.: catchorg/Catch2#2221 catchorg/Catch2@581c46249acf8389e9 We follow suit, and our JUnit XML output now validates against both the Jenkins JUnit and xUnit plugins, as well as the original Apache Ant de-facto schema. Task-number: QTBUG-95424 Change-Id: I3097d10c03c2a29709960372301b29055d224e10 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> (cherry picked from commit bb74e72) Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
The Jenkins xUnit plugin can't parse a junit report produced by Catch2. The time (duration) values have the wrong format. The error message is:
It seems that it expects time values only with 3 decimal digits, but the generated file contains 6.
Expected behavior
Generate a report which is parsed correctly by the xUnit / JUnit plugin.
Reproduction steps
In a jenkins build, build and invoke a Catch2 test executable with options
-r junit -o someReportFile.xml. Configure the job with the xUnit Test publisher and JUnit report source referencingsomeReportFile.xml.Platform information:
The text was updated successfully, but these errors were encountered: