Generate coverage output for Gitlab/Cobertura? #468

TheButlah · 2020-07-12T05:10:49Z

Would it be possible to add support for generating Cobertura xml coverage reports? Gitlab's coverage reporting uses Cobertura's format.

It appears as though tarpaulin supports cobertura, so it would be good if grcov could too.

The text was updated successfully, but these errors were encountered:

marco-c · 2020-07-14T13:16:33Z

Sure, I won't have time to implement it myself in the near term, but I'd be happy to review a PR to add support for it.

brycefisher · 2020-09-13T05:12:06Z

@TheButlah I just ran into the same thing myself! There is actually this Python project that converts from lcov to cobertura:
https://github.com/eriwen/lcov-to-cobertura-xml

Since I'm using the (un"slim") official Rust docker image on GitlabCI, Python 3 is already available. Right after running grcov, I added these lines in the script block, and cobertura seemed to work:

    - curl -O https://raw.githubusercontent.com/eriwen/lcov-to-cobertura-xml/master/lcov_cobertura/lcov_cobertura.py && chmod +x lcov_cobertura.py
    - ./lcov_cobertura.py lcov.info

This solves my needs for now, but is obviously less efficient than simply generating cobertura directly in grcov.

royb3 · 2020-10-27T07:55:01Z

@brycefisher Does it work fine with Gitlab CI? What about function name mangling?

brycefisher · 2020-10-27T17:24:14Z

It does!! It works great. You can see test code coverage within the merge request UI...kind of awesome!

Not my ideal solution (and adds a dependency to use Python for testing an otherwise pure Rust project), but its a decent workaround for the moment.

RE: function name mangling, I've only worked with pure Rust -- no extern "C" style FFI or anything. Do you have a good test case in mind that would verify if name mangling works out of the box?

gdesmott · 2021-01-08T14:07:34Z

Thanks a lot @brycefisher that works great!

By any chance, did you find a way to also integrate the coverage summary, see #556 ?

brycefisher · 2021-01-10T02:17:15Z

script:
    - ...
    - grcov ./grcov.zip -s . --llvm --branch --ignore-not-existing -o ./lcov.info
    - if [[ ! -e lcov_cobertura.py ]]; then curl -O https://raw.githubusercontent.com/eriwen/lcov-to-cobertura-xml/master/lcov_cobertura/lcov_cobertura.py && chmod +x lcov_cobertura.py; fi
    - ./lcov_cobertura.py lcov.info
    - grep '<coverage branch-rate="' coverage.xml | sed -e 's/.* branch-rate="\(.\)\.\(..\)\(.\).*/Test coverage \1\2.\3%/'

Uses the new llvm source-base coverage from nightly to generate coverage reports: - full html report as artifact - cobertura report for gitlab MR integration - output coverage summary for gitlab parsing Here is the regexp to set in gitlab as "Test coverage parsing": \s*lines\.*:\s*([\d\.]+%) Ignore sys crates when calculating coverage are those are fully generated anyway. Resources: - https://github.com/marco-c/rust-code-coverage-sample - mozilla/grcov#468 (comment) - https://www.greycastle.se/how-to-show-flutter-test-coverage-in-gitlab-ci/

Uses the new llvm source-base coverage from nightly to generate coverage reports: - full html report as artifact - cobertura report for gitlab MR integration - output coverage summary for gitlab parsing Here is the regexp to set in gitlab as "Test coverage parsing": \s*lines\.*:\s*([\d\.]+%) Resources: - https://github.com/marco-c/rust-code-coverage-sample - mozilla/grcov#468 (comment) - https://www.greycastle.se/how-to-show-flutter-test-coverage-in-gitlab-ci/

seanpianka · 2021-03-07T22:07:18Z

@brycefisher When I use the python (3) script on my lcov.info output, I get this error message...

$ grcov "${ZIP_NAME}" \
    --source-dir . \
    --binary-path ./target/debug/ \
    --output-path "${COVERAGE_OUTPUT_PATH}" \
    --branch \
    --llvm \
    --ignore-not-existing \
    --keep-only "mycoolcrate-*" 
$ scripts/lcov_cobertura.py "${COVERAGE_OUTPUT_PATH}"
Traceback (most recent call last):
  File "scripts/lcov_cobertura.py", line 416, in <module>
    main()
  File "scripts/lcov_cobertura.py", line 409, in main
    cobertura_xml = lcov_cobertura.convert()
  File "scripts/lcov_cobertura.py", line 87, in convert
    coverage_data = self.parse()
  File "scripts/lcov_cobertura.py", line 197, in parse
    function_line, function_name = line_parts[-1].strip().split(',')
ValueError: too many values to unpack (expected 2)

Any advice on what could be wrong here? Thanks.

Edit: Here's a fix... only splitting once for FN and FNDA:

diff --git a/scripts/lcov_cobertura.py b/scripts/lcov_cobertura.py
index affef27a..0c07b2af 100755
--- a/scripts/lcov_cobertura.py
+++ b/scripts/lcov_cobertura.py
@@ -194,11 +194,11 @@ class LcovCobertura(object):
                 file_branches_covered = int(line_parts[1])
             elif input_type == 'FN':
                 # FN:5,(anonymous_1)
-                function_line, function_name = line_parts[-1].strip().split(',')
+                function_line, function_name = line_parts[-1].strip().split(',', 1)
                 file_methods[function_name] = [function_line, '0']
             elif input_type == 'FNDA':
                 # FNDA:0,(anonymous_1)
-                (function_hits, function_name) = line_parts[-1].strip().split(',')
+                (function_hits, function_name) = line_parts[-1].strip().split(',', 1)
                 if function_name not in file_methods:
                     file_methods[function_name] = ['0', '0']
                 file_methods[function_name][-1] = function_hits

brycefisher · 2021-03-08T21:57:04Z

Cool! My original snippet pulled from master on the Python project, so either it never worked for your use case, or an update since I posted broke it, or something along those lines. Sometimes, pinning a version can be handy for making things continue to work the same way over time (that or making you into a bitrotten purgatory that never ends...YMMV)

seanpianka · 2021-03-08T22:49:38Z

Yep, thanks for mentioning that conversion script! Could you help with an issue related to GitLab?

I export the generated coverage.xml report, but GitLab seems to do nothing with it... I don't see anything related to code-coverage in the merge request, no results or line highlighting.

all:
  extends:
    - .unit-test
  script:
    # Generate Cobertura report
    - grcov "${COVERAGE_ZIP_NAME}" \
        --source-dir . \
        --binary-path ./target/debug/ \
        --output-path "${COVERAGE_OUTPUT_PATH}/lcov.info" \
        --output-type lcov \
        --branch \
        --llvm \
        --ignore-not-existing \
        --keep-only "mycoolcrate-*"
    # Use a cargo extension to correct the code coverage results... removes extranous counts.
    - rust-covfix -o "${COVERAGE_OUTPUT_PATH}/lcov.info" "${COVERAGE_OUTPUT_PATH}/lcov.info"
    # outputs cobertura report at "./coverage.xml"
    - python3 scripts/lcov_cobertura.py "${COVERAGE_OUTPUT_PATH}/lcov.info"
  artifacts:
    reports:
      cobertura: coverage.xml

Do I need to use the grep command you posted, and use a regex to parse the code coverage from the job output?

$ grep '<coverage branch-rate="' coverage.xml | sed -e 's/.* branch-rate="\(.\)\.\(..\)\(.\).*/Test coverage \1\2.\3%/'

seanpianka · 2021-03-08T22:58:36Z

Also, is there anyway to use an lcov.info file as input to grcov -t html? I don't want grcov to regenerate the info file, I want it to re-use my processed version (the output from rust-covfix)... I can open a separate issue, if needed.

brycefisher · 2021-03-08T22:58:41Z

Okay, so there's two parts:

Feeding the full cobertura XML data into Gitlab so it can show inline test coverage on MRs. For that, the path to the coverage.xml file must be correct in the artifacts:reports:cobertura section of your .gitlabci.yml. I suspect that you might have feed the wrong path there? I generally start throwing ls and cat statements into my CI, or I download and run the gitlab runner locally to debug faster (https://bryce.fisher-fleig.org/faster-ci-debugging-with-gitlabci/ -- quite old, possibly outdated!)
Getting an overall percentage into a badge you can display on your README. That's what the grep bit does you quoted

brycefisher · 2021-03-08T23:01:18Z

Oh! One more thing, i think that you also have to have a baseline from the target branch before this stuff starts being helpful. So maybe merge a trivial branch to master, or do a MR against a feature branch. Can't remember where the docs spell that out on GitlabCI....

seanpianka · 2021-03-08T23:06:08Z

You're right, the code coverage results are visible in the merge requests! I completely overlooked those green/red lines next to each line... thanks for the tip! 😄

Uploading artifacts for successful job
Uploading artifacts...
html-coverage-report: found 503 matching files and directories 
artifact_job_id: found 1 matching files and directories 
Uploading artifacts as "archive" to coordinator... ok  id=108... responseStatus=201 Created token=5sym...
Uploading artifacts...
coverage.xml: found 1 matching files and directories 
Uploading artifacts as "cobertura" to coordinator... ok  id=108... responseStatus=201 Created token=5sym...

So, for getting the code coverage result for the badge (or somehow visible on the "Overview" page of each merge request?), I have to run the quoted grep command, then add a parsing regex to find it in the job output.

What's the parsing regex you used in your project settings? Do you think that Test coverage \d+.\d+% would work?

brycefisher · 2021-03-08T23:14:19Z

Yeah, here what I have: Test coverage (\d+\.\d)%

seanpianka · 2021-03-08T23:25:24Z

Using the quoted grep, I see a peculiar output:

+ python3 scripts/lcov_cobertura.py target/debug/coverage/lcov.info
+ grep '<coverage branch-rate="' coverage.xml
+ sed -e 's/.* branch-rate="\(.\)\.\(..\)\(.\).*/Test coverage \1\2.\3%/'
Test coverage 011.2%

Perhaps one of the capture groups is unnecessary?

seanpianka · 2021-03-08T23:43:07Z

This seems to work alright!

# raw branch-rate value
local branch_rate
branch_rate=$(grep '<coverage branch-rate="\(.*\)"' coverage.xml | sed -e 's/.* branch-rate="\(.*\)" branches-covered.*/\1/')

# multiply by 100 with bash syntax
local coverage
coverage=$(printf %.2f "$(echo "$branch_rate * 100" | bc -l)")

# format with %
echo "Test coverage ${coverage}%"

geovie · 2021-04-13T07:27:30Z

FYI: #599 tries to implement this for grcov.

Please give it a try 🙏

Uses the new llvm source-base coverage from nightly to generate coverage reports: - full html report as artifact - cobertura report for gitlab MR integration - output coverage summary for gitlab parsing Here is the regexp to set in gitlab as "Test coverage parsing": \s*lines\.*:\s*([\d\.]+%) Resources: - https://github.com/marco-c/rust-code-coverage-sample - mozilla/grcov#468 (comment) - https://www.greycastle.se/how-to-show-flutter-test-coverage-in-gitlab-ci/

Fixes #468

marco-c · 2021-04-20T19:30:04Z

Cobertura output was implemented, it'd be great if one of you could add some docs in README.md to explain how to use it in GitLab (just like the "grcov with Travis" section).

marco-c added the enhancement label Jul 14, 2020

marco-c mentioned this issue Jan 21, 2021

output coverage summary (gitlab integration) #556

Open

marco-c linked a pull request Jan 21, 2021 that will close this issue

Add gitlab config #487

Closed

marco-c removed a link to a pull request Jan 21, 2021

Add gitlab config #487

Closed

geovie mentioned this issue Apr 9, 2021

Add cobertura output option #599

Merged

marco-c closed this as completed in #599 Apr 19, 2021

marco-c pushed a commit that referenced this issue Apr 19, 2021

Add cobertura output option (#599)

61d92d2

Fixes #468

fh1ch mentioned this issue Jan 23, 2022

Add all lines to class coverage #759

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate coverage output for Gitlab/Cobertura? #468

Generate coverage output for Gitlab/Cobertura? #468

TheButlah commented Jul 12, 2020 •

edited

marco-c commented Jul 14, 2020

brycefisher commented Sep 13, 2020

royb3 commented Oct 27, 2020

brycefisher commented Oct 27, 2020

gdesmott commented Jan 8, 2021

brycefisher commented Jan 10, 2021

seanpianka commented Mar 7, 2021 •

edited

brycefisher commented Mar 8, 2021

seanpianka commented Mar 8, 2021 •

edited

seanpianka commented Mar 8, 2021

brycefisher commented Mar 8, 2021

brycefisher commented Mar 8, 2021

seanpianka commented Mar 8, 2021 •

edited

brycefisher commented Mar 8, 2021

seanpianka commented Mar 8, 2021 •

edited

seanpianka commented Mar 8, 2021 •

edited

geovie commented Apr 13, 2021

marco-c commented Apr 20, 2021

Generate coverage output for Gitlab/Cobertura? #468

Generate coverage output for Gitlab/Cobertura? #468

Comments

TheButlah commented Jul 12, 2020 • edited

marco-c commented Jul 14, 2020

brycefisher commented Sep 13, 2020

royb3 commented Oct 27, 2020

brycefisher commented Oct 27, 2020

gdesmott commented Jan 8, 2021

brycefisher commented Jan 10, 2021

seanpianka commented Mar 7, 2021 • edited

brycefisher commented Mar 8, 2021

seanpianka commented Mar 8, 2021 • edited

seanpianka commented Mar 8, 2021

brycefisher commented Mar 8, 2021

brycefisher commented Mar 8, 2021

seanpianka commented Mar 8, 2021 • edited

brycefisher commented Mar 8, 2021

seanpianka commented Mar 8, 2021 • edited

seanpianka commented Mar 8, 2021 • edited

geovie commented Apr 13, 2021

marco-c commented Apr 20, 2021

TheButlah commented Jul 12, 2020 •

edited

seanpianka commented Mar 7, 2021 •

edited

seanpianka commented Mar 8, 2021 •

edited

seanpianka commented Mar 8, 2021 •

edited

seanpianka commented Mar 8, 2021 •

edited

seanpianka commented Mar 8, 2021 •

edited