Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[cobertura] Missing branch coverage when all branches are covered #104

Open
viceice opened this issue Feb 28, 2024 · 34 comments
Open

[cobertura] Missing branch coverage when all branches are covered #104

viceice opened this issue Feb 28, 2024 · 34 comments

Comments

@viceice
Copy link

viceice commented Feb 28, 2024

I have two test projects which are running on seperate ci nodes because of different requirements.
So i get two coverage files which i like to import in jenkins reports.

The code:

else if (ambiguous && founds == 1)

Project a, which fully covers that line gives:

<line number="61" hits="1" branch="True" />

Project b doesn't cover that line (which is expected) and gives:

<line number="61" hits="0" branch="True" condition-coverage="0% (0/4)">
  <conditions>
    <condition number="0" type="jump" coverage="0%" />
    <condition number="1" type="jump" coverage="0%" />
  </conditions>
</line>

Both reports should be identical beside coverage values.
@uhafner
Copy link

uhafner commented Feb 28, 2024 •
edited

It is also worth to mention that in the same report other lines are correctly marked, e.g.:

<line number="84" hits="1" branch="True" condition-coverage="100% (2/2)">
     <conditions>
          <condition number="0" type="jump" coverage="100%" />
     </conditions>
</line>

@uhafner uhafner mentioned this issue Feb 28, 2024
Merged
@jakubch1
Copy link
Member

@viceice could you please upgrade all your test projects to use latest Microsoft.NET.Test.Sdk and Microsoft.CodeCoverage. What commands you use to generate reports?

@viceice
Copy link
Author

viceice commented Feb 29, 2024

@viceice could you please upgrade all your test projects to use latest Microsoft.NET.Test.Sdk and Microsoft.CodeCoverage.

@jakubch1 I'm using the latest non-preview (17.9.0) of Microsoft.NET.Test.Sdk which has a direct dependency to Microsoft.CodeCoverage v17.9.0.

I'm using lockfiles and central versioning, so versions are verified. Should i try the latest preview?

What commands you use to generate reports?

I'm using a runsettings file:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <RunConfiguration>
    <TargetPlatform>x64</TargetPlatform>
  </RunConfiguration>

  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0">
        <Configuration>
          <Format>Cobertura</Format>
          <CodeCoverage>
            <ModulePaths>
              <!-- Disable native instrumentation, causes SharePoint debug issues -->
              <EnableStaticNativeInstrumentation>False</EnableStaticNativeInstrumentation>
              <EnableDynamicNativeInstrumentation>False</EnableDynamicNativeInstrumentation>
              <Include>
                <ModulePath>Proj(\.[.a-zA-Z]+)?\.dll$</ModulePath>
              </Include>
              <Exclude>
                <ModulePath>Proj(\..+)?\.Test\.dll$</ModulePath>
              </Exclude>
            </ModulePaths>
            <Attributes>
              <Exclude>
                <Attribute>^System\.CodeDom\.Compiler\.GeneratedCodeAttribute$</Attribute>
                <Attribute>^System\.Diagnostics\.CodeAnalysis\.ExcludeFromCodeCoverageAttribute$</Attribute>
              </Exclude>
            </Attributes>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

  <NUnit>
    <TestOutputXml>nunit</TestOutputXml>
    <OutputXmlFolderMode>RelativeToResultDirectory</OutputXmlFolderMode>
    <NewOutputXmlFileForEachRun>true</NewOutputXmlFileForEachRun>
  </NUnit>
</RunSettings>

which is referenced in repo root Directory.Build.props like this:

  <RunSettingsFilePath>$(MSBuildThisFileDirectory)/proj.runsettings</RunSettingsFilePath>

@jakubch1
Copy link
Member

@viceice thanks for info. How you execute tests? dotnet test on solution level? Could you please post output of those operations?

@viceice
Copy link
Author

viceice commented Feb 29, 2024 •
edited

I'm running two comands on CI.

First running cheap tests in ./test folder on a general jenkins node inside a docker windows 2022 container. There is a sln and some test projects.

dotnet test test --no-build
...
Passed!  - Failed:     0, Passed:   680, Skipped:    14, Total:   694, Duration: 17 s - Proj.Test.dll (net8.0)

Attachments:
  ~\proj\test\Proj.XY.Test\TestResults\c64ddcc7-72a7-45a8-aae4-32e5cafbb395\user_PCNAME_2024-02-29.08_36_39.cobertura.xml

This gives the first coverage file.

The second command runs on a native SharePoint server for CI (jenkins node) purposes.
So we only run SharePoint tests here. There is a sln and one test projects.

dotnet test sp/test --no-build
...
Passed!  - Failed:     0, Passed:   680, Skipped:    14, Total:   694, Duration: 17 s - Proj.SP.Test.dll (net48)

Attachments:
  ~\proj\sp\test\Proj.SP.Test\TestResults\5f49f3d0-341e-4fe5-83ac-e6e2bd5f991e\user_PCNAME_2024-02-29.08_37_31.cobertura.xml

@jakubch1
Copy link
Member

@viceice thanks for info. What version of .NET SDK you use?

@fhnaseer I suspect that that dotnet test test --no-build is executing several tests projects and doing at the end attachments merging. Could you please check which version of merging is used here? I remember some old merging was dropping conditions.

@viceice
Copy link
Author

viceice commented Feb 29, 2024

my local system

> dotnet --info                                                                                                                        ─╯
.NET SDK:
 Version:           8.0.200
 Commit:            438cab6a9d
 Workload version:  8.0.200-manifests.91505830

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22631
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.200\

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.2
  Architecture: x64
  Commit:       1381d5ebd2

.NET SDKs installed:
  6.0.419 [C:\Program Files\dotnet\sdk]
  7.0.406 [C:\Program Files\dotnet\sdk]
  8.0.102 [C:\Program Files\dotnet\sdk]
  8.0.200 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Can't check CI right now, but can reproduce locally with above versions.

I've also Visual Studio 17.9.1 on my local machine and the corresponding build tools on CI.

@jakubch1
Copy link
Member

@viceice thanks for reporting this. We will work on fixing it when using dotnet test. As workaround you could try to switch to use dotnet-coverage tool during test run:

dotnet-coverage collect -s coverage.config "dotnet test test --no-build"

You would need also to remove runsettings file from csproj and convert it to coverage.config (main tag should be <Configuration>).

@viceice
Copy link
Author

viceice commented Feb 29, 2024

my current workaround is to use dotnet coverage merge to merge both reports, which can then imported to jenkins.

@viceice
Copy link
Author

viceice commented Feb 29, 2024

is there any other benefit to use dotnet coverage over build-in ?

@jakubch1
Copy link
Member

By using dotnet-coverage you get better performance and always latest features.

Build-in tool is running coverage for each test project separately and then merges results. While dotnet-coverage is using 1 collection for all test projects and merging is not needed. It means that if same project is referenced in many test projects build-in needs to instrument it several times. External tool instruments just once. Also external tool targets .net6 and is more optimized.

@viceice
Copy link
Author

viceice commented Feb 29, 2024

ok, will give it a try when our infra is back running 🙈

neverless this bug should be fixed anyways 🙃

@viceice
Copy link
Author

viceice commented Feb 29, 2024 •
edited

OK, dotnet-coverage works partly better. I now have totally missing lines in one report.
they are condittional net8.0 only and can't be coverred by net48 of cause:

#if NETFRAMEWORK
// some full framework code
#else
// some dotnet core code
#endif

So the lines for the non-full framework are missing in the sp report.

@fhnaseer
Copy link
Member

fhnaseer commented Mar 1, 2024

@viceice Can you please tell version of dotnet-coverage?

@viceice
Copy link
Author

viceice commented Mar 1, 2024 •
edited

@viceice Can you please tell version of dotnet-coverage?

still same 17.9.0 as before

it's 17.10.3

@fhnaseer
Copy link
Member

fhnaseer commented Mar 4, 2024

I am looking at the Jenkins ticket. With the latest dotnet-coverage merge it is working as expected, right?

@jakubch1
Copy link
Member

jakubch1 commented Mar 4, 2024

@fhnaseer original issue (missing conditions) is not when merging is done by dotnet-coverage. I think case here is when there is attachments processing (merging) done by TP/.NET SDK.

@viceice can you please list all versions of Microsoft.NET.Test.Sdk for all projects?

@viceice
Copy link
Author

viceice commented Mar 4, 2024

it's same as already mentioned above. I'm using central versioning and lockfiles. I also verified before.

@fhnaseer
Copy link
Member

fhnaseer commented Mar 6, 2024

I'm running two comands on CI.

First running cheap tests in ./test folder on a general jenkins node inside a docker windows 2022 container. There is a sln and some test projects.

dotnet test test --no-build
...
Passed!  - Failed:     0, Passed:   680, Skipped:    14, Total:   694, Duration: 17 s - Proj.Test.dll (net8.0)

Attachments:
  ~\proj\test\Proj.XY.Test\TestResults\c64ddcc7-72a7-45a8-aae4-32e5cafbb395\user_PCNAME_2024-02-29.08_36_39.cobertura.xml

This gives the first coverage file.

The second command runs on a native SharePoint server for CI (jenkins node) purposes. So we only run SharePoint tests here. There is a sln and one test projects.

dotnet test sp/test --no-build
...
Passed!  - Failed:     0, Passed:   680, Skipped:    14, Total:   694, Duration: 17 s - Proj.SP.Test.dll (net48)

Attachments:
  ~\proj\sp\test\Proj.SP.Test\TestResults\5f49f3d0-341e-4fe5-83ac-e6e2bd5f991e\user_PCNAME_2024-02-29.08_37_31.cobertura.xml

@viceice Can you please run test on individual test projects (rather on solution level) and share coverage files generated per test project?

@viceice
Copy link
Author

viceice commented Mar 6, 2024

I can only provide parts of it. hopefully tomorrow.

I think the lines already mentioned are the interesting ones.

@jakubch1
Copy link
Member

jakubch1 commented Mar 8, 2024

@viceice we believe what is happening is your case:

  1. When you run dotnet test test --no-build you execute tests for several projects and our internal logic is generating coverage report for each project separately (with conditions, you don't even see those reports on output)
  2. Our logic is invoking merging of all reports from point 1. As logic is not able to correctly merge conditions from all reports => conditions are dropped. This issue is explained here: dotnet-coverage merge branch coverage issues + missing branch attributes #11 (comment) Unfortunately because of not full information is available in cobertura report, conditions have to be dropped in certain scenarios.

To make sure that this is your scenario we need to get information about line 61 for all projects from point 1 (separately) and if this is the scenario we will not be able to fix it. The only way is to use dotnet-coverage tool which has a little bit different architecture and is able to merge all conditions as it contains additional information during merge.

@viceice
Copy link
Author

viceice commented Mar 11, 2024

Report a

          <lines>
            <line number="15" hits="1" branch="False" />
            <line number="21" hits="1" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="1" branch="False" />
            <line number="28" hits="1" branch="False" />
            <line number="29" hits="1" branch="False" />
            <line number="30" hits="1" branch="False" />
            <line number="31" hits="1" branch="False" />
            <line number="32" hits="1" branch="False" />
            <line number="33" hits="1" branch="False" />
            <line number="34" hits="1" branch="False" />
            <line number="37" hits="1" branch="False" />
            <line number="38" hits="1" branch="True" />
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="1" branch="False" />
            <line number="45" hits="1" branch="True" condition-coverage="100% (4/4)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
                <condition number="1" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="46" hits="1" branch="False" />
            <line number="47" hits="1" branch="False" />
            <line number="48" hits="1" branch="False" />
            <line number="50" hits="1" branch="True" condition-coverage="100% (2/2)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="51" hits="1" branch="False" />
            <line number="52" hits="1" branch="False" />
            <line number="54" hits="1" branch="False" />
            <line number="55" hits="1" branch="True" />
            <line number="56" hits="1" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="1" branch="True" />
            <line number="62" hits="1" branch="False" />
            <line number="63" hits="1" branch="False" />
            <line number="64" hits="1" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="1" branch="False" />
            <line number="71" hits="1" branch="False" />
            <line number="73" hits="1" branch="False" />
          </lines>

Report b

          <lines>
            <line number="15" hits="0" branch="False" />
            <line number="21" hits="0" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="0" branch="False" />
            <line number="28" hits="0" branch="False" />
            <line number="29" hits="0" branch="False" />
            <line number="30" hits="0" branch="False" />
            <line number="31" hits="0" branch="False" />
            <line number="32" hits="0" branch="False" />
            <line number="33" hits="0" branch="False" />
            <line number="34" hits="0" branch="False" />
            <line number="37" hits="0" branch="False" />
            <line number="38" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="0" branch="False" />
            <line number="45" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="46" hits="0" branch="False" />
            <line number="47" hits="0" branch="False" />
            <line number="48" hits="0" branch="False" />
            <line number="50" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="51" hits="0" branch="False" />
            <line number="52" hits="0" branch="False" />
            <line number="54" hits="0" branch="False" />
            <line number="55" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="56" hits="0" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="62" hits="0" branch="False" />
            <line number="63" hits="0" branch="False" />
            <line number="64" hits="0" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="0" branch="False" />
            <line number="71" hits="0" branch="False" />
            <line number="73" hits="0" branch="False" />
          </lines>

Report c

          <lines>
            <line number="15" hits="0" branch="False" />
            <line number="21" hits="0" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="0" branch="False" />
            <line number="28" hits="0" branch="False" />
            <line number="29" hits="0" branch="False" />
            <line number="30" hits="0" branch="False" />
            <line number="31" hits="0" branch="False" />
            <line number="32" hits="0" branch="False" />
            <line number="33" hits="0" branch="False" />
            <line number="34" hits="0" branch="False" />
            <line number="37" hits="0" branch="False" />
            <line number="38" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="0" branch="False" />
            <line number="45" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="46" hits="0" branch="False" />
            <line number="47" hits="0" branch="False" />
            <line number="48" hits="0" branch="False" />
            <line number="50" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="51" hits="0" branch="False" />
            <line number="52" hits="0" branch="False" />
            <line number="54" hits="0" branch="False" />
            <line number="55" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="56" hits="0" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="62" hits="0" branch="False" />
            <line number="63" hits="0" branch="False" />
            <line number="64" hits="0" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="0" branch="False" />
            <line number="71" hits="0" branch="False" />
            <line number="73" hits="0" branch="False" />
          </lines>

Report d

          <lines>
            <line number="15" hits="0" branch="False" />
            <line number="21" hits="0" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="0" branch="False" />
            <line number="28" hits="0" branch="False" />
            <line number="29" hits="0" branch="False" />
            <line number="30" hits="0" branch="False" />
            <line number="31" hits="0" branch="False" />
            <line number="32" hits="0" branch="False" />
            <line number="33" hits="0" branch="False" />
            <line number="34" hits="0" branch="False" />
            <line number="37" hits="0" branch="False" />
            <line number="38" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="0" branch="False" />
            <line number="45" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="46" hits="0" branch="False" />
            <line number="47" hits="0" branch="False" />
            <line number="48" hits="0" branch="False" />
            <line number="50" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="51" hits="0" branch="False" />
            <line number="52" hits="0" branch="False" />
            <line number="54" hits="0" branch="False" />
            <line number="55" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="56" hits="0" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="62" hits="0" branch="False" />
            <line number="63" hits="0" branch="False" />
            <line number="64" hits="0" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="0" branch="False" />
            <line number="71" hits="0" branch="False" />
            <line number="73" hits="0" branch="False" />
          </lines>

@viceice
Copy link
Author

viceice commented Mar 11, 2024

Only project a fully covers the line 61 (see code in issue description)

@jakubch1
Copy link
Member

Is project a single test project or solution? Could you please provide diagnostic logs from project a? https://github.com/microsoft/vstest/blob/main/docs/troubleshooting.md#collect-logs-and-crash-dump

@viceice
Copy link
Author

viceice commented Mar 12, 2024

Is project a single test project or solution? Could you please provide diagnostic logs from project a?

Yes, there are 4 test projects, but they depend on each other (tree)

https://github.com/microsoft/vstest/blob/main/docs/troubleshooting.md#collect-logs-and-crash-dump
Will try todo when i've some time. Will there be our private source code inside those dumps?

@jakubch1
Copy link
Member

You can skip dumps. Enough for us is: --diag:log.txt

@viceice
Copy link
Author

viceice commented Mar 12, 2024

wait, that project build's for net8.0 and net48. so probably another coverage result merge happens?

now running for both frameworks seperately.

@viceice
Copy link
Author

viceice commented Mar 12, 2024

Project a with net8.0

          <lines>
            <line number="15" hits="1" branch="False" />
            <line number="21" hits="1" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="1" branch="False" />
            <line number="28" hits="1" branch="False" />
            <line number="29" hits="1" branch="False" />
            <line number="30" hits="1" branch="False" />
            <line number="31" hits="1" branch="False" />
            <line number="32" hits="1" branch="False" />
            <line number="33" hits="1" branch="False" />
            <line number="34" hits="1" branch="False" />
            <line number="37" hits="1" branch="False" />
            <line number="38" hits="1" branch="True" condition-coverage="50% (1/2)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="1" branch="False" />
            <line number="45" hits="1" branch="True" condition-coverage="100% (4/4)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
                <condition number="1" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="46" hits="1" branch="False" />
            <line number="47" hits="1" branch="False" />
            <line number="48" hits="1" branch="False" />
            <line number="50" hits="1" branch="True" condition-coverage="100% (2/2)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="51" hits="1" branch="False" />
            <line number="52" hits="1" branch="False" />
            <line number="54" hits="1" branch="False" />
            <line number="55" hits="1" branch="True" condition-coverage="50% (1/2)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="56" hits="1" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="1" branch="True" condition-coverage="50% (2/4)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
                <condition number="1" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="62" hits="1" branch="False" />
            <line number="63" hits="1" branch="False" />
            <line number="64" hits="1" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="1" branch="False" />
            <line number="71" hits="1" branch="False" />
            <line number="73" hits="1" branch="False" />
          </lines>

Project a with net48

          <lines>
            <line number="15" hits="1" branch="False" />
            <line number="21" hits="1" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="1" branch="False" />
            <line number="28" hits="1" branch="False" />
            <line number="29" hits="1" branch="False" />
            <line number="30" hits="1" branch="False" />
            <line number="31" hits="1" branch="False" />
            <line number="32" hits="1" branch="False" />
            <line number="33" hits="1" branch="False" />
            <line number="34" hits="1" branch="False" />
            <line number="37" hits="1" branch="False" />
            <line number="38" hits="1" branch="True" condition-coverage="50% (1/2)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="1" branch="False" />
            <line number="45" hits="1" branch="True" condition-coverage="100% (4/4)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
                <condition number="1" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="46" hits="1" branch="False" />
            <line number="47" hits="1" branch="False" />
            <line number="48" hits="1" branch="False" />
            <line number="50" hits="1" branch="True" condition-coverage="100% (2/2)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="51" hits="1" branch="False" />
            <line number="52" hits="1" branch="False" />
            <line number="54" hits="1" branch="False" />
            <line number="55" hits="1" branch="True" condition-coverage="50% (1/2)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="56" hits="1" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="1" branch="True" condition-coverage="50% (2/4)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
                <condition number="1" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="62" hits="1" branch="False" />
            <line number="63" hits="1" branch="False" />
            <line number="64" hits="1" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="1" branch="False" />
            <line number="71" hits="1" branch="False" />
            <line number="73" hits="1" branch="False" />
          </lines>

@viceice
Copy link
Author

viceice commented Mar 12, 2024

So it seems it's again the wrong merge when running both framework tests once 🤔

@jakubch1
Copy link
Member

Unfortunately this can't be fixed because:

            <line number="61" hits="1" branch="True" condition-coverage="50% (2/4)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
                <condition number="1" type="jump" coverage="50%" />
              </conditions>
            </line>

means line has 2 conditions and for each one case is covered. We are not able to calculate correct merge result (we don't know which part of condition was really covered) so we decide to remove conditions (similar to ReportGenerator). It is limitation of cobertura format.

50% + 50% could be merged to 50% or 100%.

The only workaround is to use dotnet-coverage collect to execute all your tests. In this scenario we keep internally additional data (which exactly part of condition was covered) and we are able to perform full merge.

@viceice
Copy link
Author

viceice commented Mar 12, 2024 •
edited

Thanks, i already use dotnet-coverage collect now on ci runs and the normal collector inside visual studio to see coverage there (Fine Code Coverage plugin).

@uhafner
Copy link

uhafner commented Mar 12, 2024

Unfortunately this can't be fixed because:

            <line number="61" hits="1" branch="True" condition-coverage="50% (2/4)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
                <condition number="1" type="jump" coverage="50%" />
              </conditions>
            </line>

means line has 2 conditions and for each one case is covered. We are not able to calculate correct merge result (we don't know which part of condition was really covered) so we decide to remove conditions (similar to ReportGenerator). It is limitation of cobertura format.

50% + 50% could be merged to 50% or 100%.

The only workaround is to use dotnet-coverage collect to execute all your tests. In this scenario we keep internally additional data (which exactly part of condition was covered) and we are able to perform full merge.

I understand that this is a limitation of the cobertura report format (or more precisely this is a problem of all reports, the actual coverage can only be computed with the execution data).

Despite of this architectural problem I think it would be better if you change the output in such cases. Your current output is brach=true without a condition-coverage. That means that the condition coverage will be automatically set to 100% (see DTD). I think it would be better to set in this case also the condition-coverage="100% (4/4) explicitly. Then other tools can report at least the correct number of branches (if covered or not is undefined, so the <conditions> can be skipped. (Another option would be to use the minimum coverage, so 50%)

@jakubch1 jakubch1 reopened this Mar 12, 2024
@viceice viceice mentioned this issue Mar 15, 2024
Closed
@baumheld
Copy link 

  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0">
        <Configuration>
          <Format>Cobertura</Format>
          <CodeCoverage>
            <ModulePaths>
              <!-- Disable native instrumentation, causes SharePoint debug issues -->
              <EnableStaticNativeInstrumentation>False</EnableStaticNativeInstrumentation>
              <EnableDynamicNativeInstrumentation>False</EnableDynamicNativeInstrumentation>
            </ModulePaths>
            [...]
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

@viceice Just a heads up. Are you sure <EnableStaticNativeInstrumentation> and <EnableDynamicNativeInstrumentation> should be under the node <ModulePaths>? Shouldn't it be one level up, under <CodeCoverage> according to these docs?

@viceice
Copy link
Author

viceice commented Mar 30, 2024

no, was a typo and already moved to correct place

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@uhafner @fhnaseer @viceice @jakubch1 @baumheld