MergeWith cobertura.xml make error

[tiber76]

Hi,

I need to use /p:MergeWith to merge differents results with dotnet test with coverlet.

I use coverlet.msbuild : 2.3.2 ( but i have the same error with the last version 2.4.0 )

My command :
dotnet test "C:\Users\Tiber\source\repos\LH.api\LH.Common\Tests\LH.Common.Repository.UnitTest\LH.Common.Repository.UnitTest.csproj" /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput="C:\Users\Tiber\source\repos\LH.api\LH.Common\TestResults\CodeCoverage" /p:MergeWith="C:\Users\Tiber\source\repos\LH.api\LH.Common\TestResults\CodeCoverage\coverage.cobertura.xml" --configuration Release

My First test is done without error and generate result file here :
C:\Users\Tiber\source\repos\LH.api\LH.Common\TestResults\CodeCoverage\coverage.cobertura.xml

But my second test ( to merge with the first ) fail with the same command, see error below :

Thank's for support.

Jérémy

The value given to /p:MergeWith must be a path to Coverlet's own json result format.

There hasn't been a release of Coverlet since this feature was added, so for now you'd have to merge all of your results as json, and then convert it to the format you want.

[tiber76]

Humm ok thank's ! But how to convert immediatly this json to cobertura.xml ? do i must to convert this file manually ?

[tonerdo]

Hi @tiber76, like @vlef said, you can only merge with Coverlet's own format. Your first run should generate the regular Coverlet json and your second run should merge with the json output of the previous run and set your output to cobertura

[tiber76]

Thank you Very much :) And Thank for your tool 👍

[farzadmf]

For future visitors, this SO answer was very useful, which basically removes the redundant step of running the tests for a second time.
Basically, the solution is to output two formats, something like this in bash:

dotnet test <solution-file> \
  # ...
  /p:CoverletOutputFormat=\"json,cobertura\" \
  /p:MergeWith=<result-dir>/coverage/coverage.json

[MarcoRossignoli]

@farzadmf it works only if you run test sequentially, check this thread #598 (comment)

[farzadmf]

Oh wow @MarcoRossignoli , that's a good point (and a tricky one)
Just wondering: isn't the sequential (or parallel) run a potential problem with mergeWith anyways?

[MarcoRossignoli]

With sequential execution you'll append one result at time so at the end you'll have all results merged, very similar to https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples.md#msbuild-integration

I mean json will be merged with previous and so you'll overwrite will all tests history but with no possible overlapping

[MarcoRossignoli]

BTW the "by design" usage should be run one dotnet test for every test project and merge at the end, I know it's not great from usage perspective, but msbuild engine doesn't handle solutions (sln) but projects(csproj) so at the moment there is no great way to synch parallel project builds/tests run, I mean when you run dotnet test on a solution actually you'll run one for every test projects in parallel, and there is no built-in way to wait all tests. That semantics is not present in msbuild.

[ckarcz]

here is a cross platform powershell script i created that finds unit test projects and generates merged coverlet format report and a final output report in cobertura format, but should work with any supported final format.

it's not perfect, but can be a good starting point for a more powerful script (checking error codes etc). it then generates a report using reportgenerator.

one minor issue i'm having is that the test result xml are overwritten since it's the same static name for each test project run. maybe this can be fixed with output file names or directories for each test project...

fwiw, i've also seen a Cake integration that may do similar to what this script does:
https://gist.github.com/ckarcz/180a9ddfd9bfe33b1e68e1634c0e599e

[MarcoRossignoli]

@ckarcz doesn't https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md works for you?

[ckarcz]

@MarcoRossignoli that would work but i cannot get msbuild to accept the multiple values for the
/p:CoverletOutputFormat parameter when executing in powershell.

i have tired all sorts of combinations of
/p:CoverletOutputFormat=\"cobertura,json\"

if you know of a way that would be most excellent 👍

[ckarcz]

@MarcoRossignoli

i have found a way around the issue i was having via this msbuild issue.

this is how you would need to do it in powershell:
/p:CoverletOutputFormat=json%2ccobertura

i have updated my gist with a comment with the improved version.

if you guys can update your documentation with a note for powershell users that may save some headaches! 😸