[feat] Add command-line option to generate a JUnit XML report

[jgphpc]

Closes #651

Example in https://git.cscs.ch/c4k/c4kcust/example-junit/-/pipelines/23226/test_report

[pep8speaks]

Hello @jgphpc, Thank you for updating!

Cheers! There are no PEP8 issues in this Pull Request!Do see the ReFrame Coding Style Guide

Comment last updated at 2021-05-04 20:48:08 UTC

[teojgo]

I would say to either have a separate option for the xml report or choose the format based on the extension of the filename passed by the user. Also, it could be nice if we could somehow pretty print the xml with proper indentation.

[jgphpc]

I would say to either have a separate option for the xml report or choose the format based on the extension of the filename passed by the user.

• I would suggest --junit-xml to be consistent with pytest
• what if you want both ?

[codecov-io]

Codecov Report

Merging #1925 (6af94ee) into master (69dff6d) will increase coverage by 0.00%.
The diff coverage is 90.90%.

@@           Coverage Diff           @@
##           master    #1925   +/-   ##
=======================================
  Coverage   87.97%   87.97%           
=======================================
  Files          50       50           
  Lines        8496     8516   +20     
=======================================
+ Hits         7474     7492   +18     
- Misses       1022     1024    +2     

Impacted Files	Coverage Δ
reframe/frontend/cli.py	75.96% <75.00%> (-0.02%)	⬇️
reframe/frontend/statistics.py	95.71% <100.00%> (+0.25%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 69dff6d...6af94ee. Read the comment docs.

[teojgo]

I would say to either have a separate option for the xml report or choose the format based on the extension of the filename passed by the user.

• I would suggest --junit-xml to be consistent with pytest
• what if you want both ?

Ok, --junit-xml sounds good to me.

[victorusu]

@jgphpc, I am sure if this is possible, I am not familiar with JUnit, but I wonder if it makes sense and whether if this is also possible to add additional information to the JUnit... It would be nice to know the hostname. The schema that you shared in the code (https://github.com/windyroad/JUnit-Schema/blob/162a883ac631f61ca1d14a3d472e1fbb53c1c05f/JUnit.xsd#L171) shows a hostname entry.
But I wonder if we can also add additional fields...

What do you think?

[vkarak]

Example in https://git.cscs.ch/c4k/c4kcust/example-junit/-/pipelines/23226/test_report

@jgphpc Is this report generated from this PR?

[vkarak]

Also, we will need a unit test. Ideally, the unit test would be validating the generated XML against the schema. I think for validation we might need the lxml library, which also support the etree that you are using. So we could move everything over to this (it looks like an import change, but I'm not sure).

[jgphpc]

@jgphpc Is this report generated from this PR?

I updated it: https://git.cscs.ch/c4k/c4kcust/example-junit/-/pipelines/23916/test_report

[vkarak]

I updated the PR by fine tuning the implementation. It's a bit simpler now. Nonetheless, the generated XML does not validate against the schema. See the message of the failing unit tests. If we fix that and finalise the actual information of the output, it'll be ready to be merged.

Changes done and PR must be merged.