How to merge several result files together?

[AArnott]

Due to AArnott/Library.Template#2 and the various doc and product bugs linked from it, it seems the best workaround for now would be if we could merge several coverage.cobertura.xml files into just one in order to pass it to the PublishCodeCoverageResults task.

Does coverlet offer a means of doing that?

Any other suggestions?

[MarcoRossignoli]

AFAIK it's a bit tricky today...we can only merge coverlet json format...so we need to generate all json result and merge togheter and at the end merge and change format.
For coverlet repo for instance:

C:\git\coverlet (master -> origin)
λ dotnet test test\coverlet.collector.tests /p:CollectCoverage=true /p:CoverletOutput=C:\git\coverlet\resultcollectors.json
Test run for C:\git\coverlet\test\coverlet.collector.tests\bin\Debug\netcoreapp2.2\coverlet.collector.tests.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 16.1.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Test Run Successful.
Total tests: 15
     Passed: 15
 Total time: 1,9769 Seconds

Calculating coverage result...
  Generating report 'C:\git\coverlet\resultcollectors.json'

+--------------------+--------+--------+--------+
| Module             | Line   | Branch | Method |
+--------------------+--------+--------+--------+
| coverlet.collector | 74,59% | 55%    | 77,27% |
+--------------------+--------+--------+--------+
| coverlet.core      | 7,99%  | 6,49%  | 17,39% |
+--------------------+--------+--------+--------+

+---------+---------+--------+---------+
|         | Line    | Branch | Method  |
+---------+---------+--------+---------+
| Total   | 21,11%  | 12,5%  | 38,55%  |
+---------+---------+--------+---------+
| Average | 10,555% | 6,25%  | 19,275% |
+---------+---------+--------+---------+


C:\git\coverlet (master -> origin)
λ dotnet test test\coverlet.core.tests /p:CollectCoverage=true /p:CoverletOutput=C:\git\coverlet\resultcore.xml  /p:CoverletOutputFormat=cobertura /p:MergeWith=C:\git\coverlet\resultcollectors.json
Test run for C:\git\coverlet\test\coverlet.core.tests\bin\Debug\netcoreapp2.0\coverlet.core.tests.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 16.1.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:01.2539208]     Coverlet.Core.Tests.Instrumentation.ModuleTrackerTemplateTests.HitsOnMultipleThreadsCorrectlyCounted [SKIP]
  ! Coverlet.Core.Tests.Instrumentation.ModuleTrackerTemplateTests.HitsOnMultipleThreadsCorrectlyCounted [1ms]
[xUnit.net 00:00:03.5351530]     Coverlet.Core.Instrumentation.Tests.InstrumenterTests.TestCoreLibInstrumentation [SKIP]
  ! Coverlet.Core.Instrumentation.Tests.InstrumenterTests.TestCoreLibInstrumentation [1ms]

Test Run Successful.
Total tests: 114
     Passed: 112
    Skipped: 2
 Total time: 4,4710 Seconds

Calculating coverage result...
  Generating report 'C:\git\coverlet\resultcore.xml'

+--------------------+--------+--------+--------+
| Module             | Line   | Branch | Method |
+--------------------+--------+--------+--------+
| coverlet.core      | 77,12% | 67,93% | 87,57% |
+--------------------+--------+--------+--------+
| coverlet.collector | 74,59% | 55%    | 77,27% |
+--------------------+--------+--------+--------+

+---------+--------+---------+---------+
|         | Line   | Branch  | Method  |
+---------+--------+---------+---------+
| Total   | 76,62% | 66,33%  | 83,93%  |
+---------+--------+---------+---------+
| Average | 38,31% | 33,165% | 41,965% |
+---------+--------+---------+---------+

You'll need to run commands in a sequential way to avoid race.

@tonerdo do you have better solution?I'm not fluent yet on "reports strategies".

[danielpalme]

Alternative:
ReportGenerator can merge several Cobertura files and generate one result file.
It also generates HTML reports which might be useful for visualization.

[MarcoRossignoli]

Nice!
Thank's @danielpalme!

[AArnott]

Thanks!
Ironically the HTML reports produced by ReportGenerator appears to be what Azure DevOps is already using internally and showing off in the build coverage tab. It's too bad it doesn't use that tool to also merge the files for the summary tab. Maybe they'll respond to my feedback with that resolution. :)

[danielpalme]

@AArnott
Correct, Azure DevOps is already using it internally.
But you can do the following:

• Install the ReportGenerator extension: https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator
• Configure the ReportGenerator task to produce HTML and Cobertura (this is the default)
• Configure the Publish code coverage results task and point it to the HTML files directory and the merged Cobertura file generated by ReportGenerator

Currently the Publish code coverage results task regenerates the HTML report.
To avoid that, you have to create a new environment variable:
disable.coverage.autogenerate: 'true'

[AArnott]

Thanks, @danielpalme. I'm definitely seeing improvements here in AArnott/Library.Template#6 where code coverage from multiple target framework test runs is now merged in ADO. 😄

But I have a twist on the overall requirement: I run tests on 3 agents (Mac, Linux and Windows). Since ADO only shows one code coverage report, I want all these merged. I've arranged for all these to be merged by a "fan in" job in my pipeline, but the problem I'm seeing is that each agent's reports include full paths to source files, and the merging agent, while it has the source files, the paths aren't the same and the merging/report generator process warns that files are missing.

Analyzing 2 classes
Creating report 1/2 (Assembly: Library, Class: Library.Calculator)
File '/Users/vsts/agent/2.153.2/work/1/s/src/Library/Calculator.cs' does not exist (any more).
File 'D:/a/1/s/src/Library/Calculator.cs' does not exist (any more).
Creating report 2/2 (Assembly: Library, Class: ThisAssembly)
File '/home/vsts/work/1/s/obj/Library/Release/netstandard2.0/Library.Version.cs' does not exist (any more).
File '/Users/vsts/agent/2.153.2/work/1/s/obj/Library/Release/netstandard2.0/Library.Version.cs' does not exist (any more).
File 'D:/a/1/s/obj/Library/Release/netstandard2.0/Library.Version.cs' does not exist (any more).

Any suggestions?

[AArnott]

It looks like for now I'll have to settle for publishing a report from just one of the agents.

[AArnott]

Is there a way to get the report files to contain only relative paths to source files instead of absolute paths? If so, I could get this to work and merge reports across agents.

[MarcoRossignoli]

For cobertura we've an open issue #408 (comment)

Do you mean use this format https://issues.jenkins-ci.org/secure/attachment/23019/coverage.xml? Check sources and filename fields.

[AArnott]

Yes, the filename field (and I guess the sources field, although that's empty on my local machine). I'm sure absolute filenames are useful in some contexts, but if we can use relative paths then we can unify across agents.

The cobertura issue you linked to looks like it's refining absolute paths rather than switching to relative. Or am I reading it wrong?

[MarcoRossignoli]

When you say relative path...relative to what?
I mean if I understood well cobertura reports needs source+filename to get "full path"...I don't understand what you mean for "relative".
Another example https://gist.github.com/apetro/fcfffb8c4cdab2c1061d

[AArnott]

relative to what?

The repo root (or whatever I might specify). Some common root that can be reproduced across machines.

So imagine changing it like this:

-<source>/Users/apetro/code/github_jasig/uPortal/uportal-war/target/generated-sources/annotations</source>
+<source>uPortal/uportal-war/target/generated-sources/annotations</source>

Then when passing the report to the tool, /Users/apetro/code/github_jasig would either be the working directory or specified via a command line switch so that the tool knows how to create absolute paths as necessary to access the files.

Since all the reports from all agents would have relative paths, then the machine merging the reports from all of them, and producing HTML files from the source files, could find all the files it needs to, even if each report may have originally had different absolute paths.

[danielpalme]

Relative path are not necessarily easy to achieve.

In your case a script should do the job to remove the absolute path(s):

Example:

(gc "coverage.xml") | % { $_ -replace "C:\full\path", "path" } | Out-File "coverage.xml" -Encoding UTF8