The same command against the same code results different coverage values

[Andras-Csanyi]

I experience that when coverage data is collected on my build server then certain coverage is missing. The same command on my laptop includes all the coverage data. The result is validated by result of dotCover.

On my laptop the same project with the same command results this (only the important snippet):

| DigitalLibrary.MasterData.Controllers                     | 52.17% | 100%   | 50%    |

On the build server the result is the following (important snippet only)

| DigitalLibrary.MasterData.Controllers                     | 0%     | 100%   | 0%     |

I don't have any idea what can be the problem here. If any details is needed, please let me know and I'll do my best to provide it.

Project details

• project
• the tests are not collected are integration tests, the spins up a whole webhost to run tests as it is described in the MS doc
• the test cases are generated from Specflow
• here are the tests
• the test subjects are these controllers

Build server

• added as agent to Azure DevOps

Dotnet info

sayusi@ubi  ~  dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.103
 Commit:    9effbc8ad5

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  20.04
 OS Platform: Linux
 RID:         ubuntu.20.04-x64
 Base Path:   /usr/share/dotnet/sdk/5.0.103/

Host (useful for support):
  Version: 5.0.3
  Commit:  eae88cc11b

.NET SDKs installed:
  3.1.406 [/usr/share/dotnet/sdk]
  5.0.103 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 3.1.12 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.12 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Command

• the particular build can be found here

2021-02-12T19:05:13.3345252Z [command]/usr/bin/dotnet test /home/sayusi/build_agent/_work/12/s/src/DigitalLibrary.sln --logger trx --results-directory /home/sayusi/build_agent/_work/_temp --no-build --configuration Debug --verbosity normal /p:CollectCoverage=true /p:ExcludeByAttribute='Obsolete*%2cGeneratedCodeAttribute*%2cCompilerGeneratedAttribute' /p:CoverletOutputFormat=opencover /p:CoverletOutput=/home/sayusi/build_agent/_work/_temp/coverage/ /p:MergeWith=/home/sayusi/build_agent/_work/_temp/coverage/

Result

2021-02-12T19:05:52.2136081Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2137238Z | Module                                                    | Line   | Branch | Method |
2021-02-12T19:05:52.2138303Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2139050Z | DigitalLibrary.Utils.MasterDataTestHelper                 | 95.34% | 84.61% | 84.61% |
2021-02-12T19:05:52.2140200Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2140928Z | DigitalLibrary.MasterData.Ctx                             | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2141967Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2142937Z | DigitalLibrary.ControlPanel.BusinessLogic.Exceptions      | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2144216Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2145594Z | DigitalLibrary.ControlPanel.WebApi.Api                    | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2146796Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2147727Z | DigitalLibrary.MasterData.DomainModel.Interfaces          | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2148712Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2149412Z | DigitalLibrary.Utils.Guards                               | 33.33% | 34.61% | 50%    |
2021-02-12T19:05:52.2150586Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2151313Z | DigitalLibrary.MasterData.BusinessLogic.Implementations   | 37.04% | 16.93% | 42.66% |
2021-02-12T19:05:52.2153020Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2153786Z | DigitalLibrary.MasterData.DomainModel                     | 35.41% | 100%   | 36.95% |
2021-02-12T19:05:52.2154717Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2155615Z | DigitalLibrary.MasterData.BusinessLogic.Exceptions        | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2156526Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2157581Z | DigitalLibrary.MasterData.Controllers                     | 0%     | 100%   | 0%     |
2021-02-12T19:05:52.2159145Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2160006Z | DigitalLibrary.ControlPanel.Controllers                   | 0%     | 0%     | 0%     |
2021-02-12T19:05:52.2161545Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2162241Z | DigitalLibrary.MasterData.BusinessLogic.ViewModels        | 0%     | 100%   | 0%     |
2021-02-12T19:05:52.2163490Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2164273Z | DigitalLibrary.MasterData.BusinessLogic.Interfaces        | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2166455Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2168164Z | DigitalLibrary.MasterData.Validators                      | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2170097Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2171562Z | DigitalLibrary.ControlPanel.BusinessLogic.Interfaces      | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2173343Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2174731Z | DigitalLibrary.ControlPanel.WebApi.Client                 | 0%     | 0%     | 0%     |
2021-02-12T19:05:52.2176531Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2178471Z | DigitalLibrary.ControlPanel.BusinessLogic.Implementations | 0%     | 0%     | 0%     |
2021-02-12T19:05:52.2180522Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2181871Z | DigitalLibrary.ControlPanel.DomainModel                   | 0%     | 100%   | 0%     |
2021-02-12T19:05:52.2183643Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2184940Z | DigitalLibrary.MasterData.Web.Api                         | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2186629Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2187957Z | DigitalLibrary.Utils.ControlPanel.DataSample              | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2189606Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2191290Z | DigitalLibrary.Utils.IntegrationTestFactories             | 0%     | 100%   | 0%     |
2021-02-12T19:05:52.2193724Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2195282Z | DigitalLibrary.ControlPanel.Validators                    | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2197192Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2198634Z | DigitalLibrary.ControlPanel.Ctx                           | 100%   | 100%   | 100%   |
2021-02-12T19:05:52.2200571Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2202043Z | WebApp                                                    | 0%     | 0%     | 0%     |
2021-02-12T19:05:52.2204001Z +-----------------------------------------------------------+--------+--------+--------+
2021-02-12T19:05:52.2205044Z 
2021-02-12T19:05:52.2206387Z +---------+--------+--------+--------+
2021-02-12T19:05:52.2207471Z |         | Line   | Branch | Method |
2021-02-12T19:05:52.2209037Z +---------+--------+--------+--------+
2021-02-12T19:05:52.2210178Z | Total   | 27.17% | 20.47% | 31.34% |
2021-02-12T19:05:52.2211755Z +---------+--------+--------+--------+
2021-02-12T19:05:52.2212882Z | Average | 58.38% | 76.5%  | 58.92% |
2021-02-12T19:05:52.2214349Z +---------+--------+--------+--------+
2021-02-12T19:05:52.2215311Z 

My laptop

• latest OSX

Dotnet info

andrascsanyi@andras-mbp-2019  DigitalLibrary/src   feature/dimension_structure_tree  dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.102
 Commit:    71365b4d42

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  11.0
 OS Platform: Darwin
 RID:         osx.11.0-x64
 Base Path:   /usr/local/share/dotnet/sdk/5.0.102/

Host (useful for support):
  Version: 5.0.2
  Commit:  cb5f173b96

.NET SDKs installed:
  5.0.102 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Command

dotnet test --no-build \
        --configuration Debug \
        --verbosity normal \
        /p:CollectCoverage=true \
        /p:ExcludeByAttribute='Obsolete*%2cGeneratedCodeAttribute*%2cCompilerGeneratedAttribute' \
        /p:CoverletOutputFormat=opencover \
        /p:CoverletOutput='/Users/andrascsanyi/Dev/github.com/Andras-Csanyi/DigitalLibrary/src/coverage/' \
        /p:MergeWith='/Users/andrascsanyi/Dev/github.com/Andras-Csanyi/DigitalLibrary/src/coverage/'

Result:

Calculating coverage result...
  Generating report '/Users/andrascsanyi/Dev/github.com/Andras-Csanyi/DigitalLibrary/src/coverage/coverage.opencover.xml'

+-----------------------------------------------------------+--------+--------+--------+
| Module                                                    | Line   | Branch | Method |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.ControlPanel.WebApi.Client                 | 0%     | 0%     | 0%     |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.Utils.IntegrationTestFactories             | 0%     | 100%   | 0%     |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.WebApi.Client                   | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.Controllers                     | 52.17% | 100%   | 50%    |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.Utils.MasterDataTestHelper                 | 96.38% | 84.61% | 90%    |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.ControlPanel.Validators                    | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.ControlPanel.WebApi.Api                    | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.Utils.ControlPanel.DataSample              | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.ControlPanel.BusinessLogic.Interfaces      | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.ControlPanel.BusinessLogic.Exceptions      | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.BusinessLogic.ViewModels        | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.ControlPanel.Ctx                           | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.ControlPanel.DomainModel                   | 0%     | 100%   | 0%     |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.Utils.Guards                               | 16.66% | 19.23% | 33.33% |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.ControlPanel.BusinessLogic.Implementations | 0%     | 0%     | 0%     |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.DomainModel                     | 33.33% | 100%   | 50%    |
+-----------------------------------------------------------+--------+--------+--------+
| WebApp                                                    | 78.82% | 75%    | 66.66% |
+-----------------------------------------------------------+--------+--------+--------+
| DiLibHttpClientResponseObjects                            | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.Ctx                             | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.Utils.DiLibHttpClient                      | 100%   | 50%    | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.Web.Api.Client.Exceptions       | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.Web.Api.Client.Interfaces       | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.BusinessLogic.Implementations   | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.Validators                      | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.DomainModel.Interfaces          | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.BusinessLogic.Interfaces        | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.BusinessLogic.Exceptions        | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.MasterData.Web.Api                         | 100%   | 100%   | 100%   |
+-----------------------------------------------------------+--------+--------+--------+
| DigitalLibrary.ControlPanel.Controllers                   | 0%     | 100%   | 0%     |
+-----------------------------------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 78.99% | 44.28% | 69.49% |
+---------+--------+--------+--------+
| Average | 75.08% | 87.2%  | 75.51% |
+---------+--------+--------+--------+

     1>Done Building Project "/Users/andrascsanyi/Dev/github.com/Andras-Csanyi/DigitalLibrary/src/DigitalLibrary.sln" (VSTest target(s)).

[MarcoRossignoli]

Can you try with https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md

Msbuild integration suffers of a well known issue and could be related to your problem.

https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#vstest-stops-process-execution-early

[Andras-Csanyi]

Can you try with https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md

Msbuild integration suffers of a well known issue and could be related to your problem.

https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#vstest-stops-process-execution-early

If I understand correctly the description I use VSTS. Here is the project file:

• coverlet.collector package is added
• Microsoft.NET.Test.Sdk package is added
• dotnet test command is used and it is pimped a bit

Is there a difference between
dotnet test --collect:"XPlat Code Coverage"
and the command I use beside the obvious things? I mean, does --collect:"XPlat Code Coverage" enables something special my command misses?

Did I miss something?

</ItemGroup>
    <ItemGroup>
        <PackageReference Include="coverlet.collector" Version="3.0.2">
            <PrivateAssets>all</PrivateAssets>
            <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
        </PackageReference>
        <PackageReference Include="coverlet.msbuild" Version="3.0.2">
            <PrivateAssets>all</PrivateAssets>
            <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
        </PackageReference>
        <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="5.0.3"/>
        <PackageReference Include="Microsoft.CodeAnalysis.NetAnalyzers" Version="5.0.3">
            <PrivateAssets>all</PrivateAssets>
            <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
        </PackageReference>
        <PackageReference Include="SpecFlow" Version="3.6.23"/>
        <PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.6.23"/>
        <PackageReference Include="SpecFlow.xUnit" Version="3.6.23"/>
        <PackageReference Include="FluentAssertions" Version="5.10.3"/>
        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.8.3"/>
        <PackageReference Include="StyleCop.Analyzers" Version="1.1.118"/>
        <PackageReference Include="xunit" Version="2.4.1"/>
        <PackageReference Include="xunit.analyzers" Version="0.10.0"/>
        <PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">
            <PrivateAssets>all</PrivateAssets>
            <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
        </PackageReference>
    </ItemGroup>

[petli]

You need to remove the reference to coverlet.msbuild and only reference coverlet.collector. --collect:"XPlat Code Coverage" is then the flag that enables the coverage collection using the VSTest integration, so you must use that instead of /p:CollectCoverage=true.

You also need to create a runsettings file with the remaining options, as they cannot be passed on the command line to the VSTest collector. There is an example file and command line arguments on the page @MarcoRossignoli linked to.

The VSTest integration does not (yet) support merging reports (the MergeWith parameter in your command). You can instead use ReportGenerator to merge the coverage files from several coverlet runs into a single file (and also generate a HTML report at the same time).

[Andras-Csanyi]

@petli Ok, I see it. Thanks for your help. I'll take a try later today and I'll let you know.

[MarcoRossignoli]

Closing for stale conversation, feel free to reopen if needed.