Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include file path on xunit reporting #4328

Open
luiz1361 opened this issue Apr 29, 2022 · 2 comments
Open

Include file path on xunit reporting #4328

luiz1361 opened this issue Apr 29, 2022 · 2 comments

Comments

@luiz1361
Copy link

I propose the file path is included within testcases.
At the moment my use case is to use GitLab Test API exporting Robot Framework reports in junit format and scrape it to retrieve individual timings and run parallel execution of tests on several nodes.
https://docs.gitlab.com/ee/ci/unit_test_reports.html

Other test frameworks such as RSPEC would provide the file path corresponding to each test case such as below.

This is an example of report generated by RSPEC using RspecJunitFormatter

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="735" skipped="0" failures="0" errors="0" time="550.581470" timestamp="2022-04-29T00:13:24+00:00" hostname="xxx">
<properties>
<property name="seed" value="3628"/>
</properties>
<testcase classname="spec.lib.xxx" name="xxx" file="./spec/lib/xxx.rb" time="0.010554"></testcase>

This is how the GitLab Test API would present the test cases if the file was provided:

      {
        "status": "skipped",
        "name": "x",
        "classname": "X",
        "file": "x.robot",
        "execution_time": 0.001,
        "system_output": {
          "message": "Test skipped using '--skip' command line option.",
          "type": "SkipExecution"
        },
        "stack_trace": null,
        "recent_failures": null
      },
      {
        "status": "skipped",
        "name": "x",
        "classname": "X",
        "file": "x.robot",
        "execution_time": 0.001,
        "system_output": {
          "message": "Test skipped using '--skip' command line option.",
          "type": "SkipExecution"
        },
        "stack_trace": null,
        "recent_failures": null
      },
@luiz1361
Copy link
Author

#4327 opened PR

@pekkaklarck
Copy link
Member

I'm in general in favor of adding more information to the xunit output files, but because there's no official standard for this format, it's hard to know does adding something new break existing tools using xunit files. We enhanced xunit outputs in various ways in RF 5.0 and discussed about adding source and line number information as well. We even added line numbers to output.xml files (#4202) for tests to be able to add them also to xunit files, but in the end decided not to do that. I think the main problem was concern with compatibility.

The first step to get file paths to Robot's xunit files would be making sure adding them doesn't break existing tools. Testing all possible tools using these files is practically impossible, but this should be tested at least with Jenkins xunit plugin and with junitparser module. If there are no problems, I believe we can add this info already in RF 5.1. If there are severe problems after the release, we can then revert the change or make this behavior somehow configurable.

There are also two design decisions to be made:

  1. Where to add the path. The example in the description has it with each test, but I believe it would make more sense to add it to suites in our case. The reason is that all tests in a suite would have same path and its a bit stupid to repeat it. If tools actually using this information expect tests, not suites, to have the path, we can add it to tests.
  2. Decide should we add also line number information. I believe we should.

Do you @luiz1361 have interest to test does adding file and lineno attributes cause problems with Jenkins and junitparser? Could you also test would it be fine to add file to suites or should we add it to tests?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants