Code coverage results don't include net462 TFMs

[geeknoid]

In https://github.com/dotnet/extensions, the coverage report we are getting doesn't factor in data for the net462 TFM. We run our tests for a bunch of TFMs and the coverage report shows that the else branch of code that does:

#fi NETCOREAPP_3_1_OR_GREATER
// shows as covered
#else
// shows as not-covered
#endif

is not covered even though the tests definitely cover that code. I don't know if you have access to this, but here's a coverage report that shows this: https://dev.azure.com/dnceng-public/public/_build/results?buildId=336868&view=codecoverage-tab. The Redactor type in the Microsoft.Extensions.Compliance.Abstractions DLL in particular.

Is this is a known issue/limitation, or do we need to do something special to make this work?

Here's our config file in case that's the problem.

<?xml version="1.0" encoding="utf-8"?>
<Configuration>
  <Format>cobertura</Format>
  <CodeCoverage>

    <ModulePaths>
      <Include>
        <ModulePath>.*Microsoft.AspNetCore\.[^/\\]+\.dll$</ModulePath>
        <ModulePath>.*Microsoft.Extensions\.[^/\\]+\.dll$</ModulePath>
        <ModulePath>.*Microsoft.Gen\.[^/\\]+\.dll$</ModulePath>
        <ModulePath>.*System\.Cloud\.[^/\\]+\.dll$</ModulePath>
      </Include>
      <Exclude>
        <ModulePath>.*tests\.dll</ModulePath>
        <ModulePath>.*xunit.*</ModulePath>
        <ModulePath>.*moq.*</ModulePath>
      </Exclude>
    </ModulePaths>

    <!-- Match attributes on any code element: -->
    <Attributes>
      <Exclude>
        <!-- Don't forget "Attribute" at the end of the name -->
        <Attribute>^System\.Diagnostics\.DebuggerHiddenAttribute$</Attribute>
        <Attribute>^System\.Diagnostics\.DebuggerNonUserCodeAttribute$</Attribute>
        <Attribute>^System\.CodeDom\.Compiler\.GeneratedCodeAttribute$</Attribute>
        <Attribute>^System\.Diagnostics\.CodeAnalysis\.ExcludeFromCodeCoverageAttribute$</Attribute>
      </Exclude>
    </Attributes>

    <!-- When set to True, dynamic native instrumentation will be enabled. -->
    <EnableDynamicNativeInstrumentation>false</EnableDynamicNativeInstrumentation>
    <!-- When set to True, instrumented binaries on disk are removed and original files are restored. -->
    <EnableStaticNativeInstrumentationRestore>false</EnableStaticNativeInstrumentationRestore>
    <EnableStaticNativeInstrumentation>false</EnableStaticNativeInstrumentation>
    <UseVerifiableInstrumentation>false</UseVerifiableInstrumentation>

  </CodeCoverage>
</Configuration>

Thanks.

@RussKie

AB#1866670

[fhnaseer]

@geeknoid Can you please add -l coverage.log -ll Verbose arguments to dotnet-coverage collect and share log file?

[geeknoid]

@RussKie Can you follow-up on this for me please?

[RussKie]

@geeknoid Martin Taillefer FTE Can you please add -l coverage.log -ll Verbose arguments to dotnet-coverage collect and share log file?

I added these settings and ran a build. You can find the logs here:

[fhnaseer]

I looked into the logs and coverage files. For the coverage file generated against windows, hits=0 for net462 lines.

<line number="42" hits="1" branch="False" />
<line number="43" hits="1" branch="False" />
<line number="48" hits="0" branch="False" />
<line number="52" hits="0" branch="False" />
<line number="53" hits="0" branch="False" />
<line number="55" hits="0" branch="False" />
<line number="59" hits="0" branch="False" />
<line number="60" hits="0" branch="False" />
<line number="62" hits="0" branch="False" />

I tried to repro this locally, but I was able to get coverage correctly.

<line number="42" hits="1" branch="False" />
<line number="43" hits="1" branch="False" />
<line number="48" hits="1" branch="False" />
<line number="52" hits="1" branch="False" />
<line number="53" hits="1" branch="False" />
<line number="55" hits="1" branch="False" />
<line number="59" hits="1" branch="False" />
<line number="60" hits="1" branch="False" />
<line number="62" hits="1" branch="False" />

I see that you are using dotnet-coverage 17.7.1 version. I would suggest using latest version 17.8.0 and check the results.

[RussKie]

I updated the tool to 17.8 and re-ran the build. The logs are here: https://dev.azure.com/dnceng-public/public/_build/results?buildId=344356&view=artifacts&pathAsName=false&type=publishedArtifacts

[RussKie]

@geeknoid which API can we look at to verify to coverage?

[geeknoid]

You can look at Microsoft.Extensions.Compliance.Abstractions. Sadly, using the latest version didn't help.

[fhnaseer]

Can we do two more things?

1. Run tests only for Microsoft.Extensions.Compliance.Abstractions test (only net462) and see results.
2. Run tests only for Microsoft.Extensions.Compliance.Abstractions test (all TFMs) and see results.

[RussKie]

Here's a build that only changes dotnet-coverage version to 17.8.0. Microsoft.Extensions.Compliance.Abstractions coverage is reported at 91.1%.
This build is on top of the previous and only builds and tests Microsoft.Extensions.Compliance.Abstractions. The coverage is reported at 100%. The logs are available as well:

@fhnaseer I'm going OOF, but you can to repro this locally:

Local repro steps

To build and test only Microsoft.Extensions.Compliance.Abstractions for all TFMs:

git clone https://github.com/dotnet/extensions.git test
cd test
git switch RussKie-patch-1
./build.cmd -vs Microsoft.Extensions.Compliance.Abstractions -nolaunch
./build.cmd -c Release -restore -build -testCoverage

Only test for a specific TFM you need to tweak the build script:

diff --git a/eng/build.ps1 b/eng/build.ps1
index f92c89ed5a..e7072dbbd8 100644
--- a/eng/build.ps1
+++ b/eng/build.ps1
@@ -196,7 +196,7 @@ if ($testCoverage) {
     $testResultPath = "./artifacts/TestResults/$configuration";
 
     # Run tests and collect code coverage
-    ./.dotnet/dotnet dotnet-coverage collect --settings ./eng/CodeCoverage.config --output $testResultPath/local.cobertura.xml "build.cmd -test -configuration $configuration -bl:`$$binaryLog"
+    ./.dotnet/dotnet dotnet-coverage collect --settings ./eng/CodeCoverage.config --output $testResultPath/local.cobertura.xml "build.cmd -test -configuration $configuration /p:TestTargetFrameworks=net462 -bl:`$$binaryLog"

The run again (no need to restore or re-build again):

./build.cmd -c Release -testCoverage

[fhnaseer]

@RussKie I tried to repro it. I found out that tests are not executed for net462. Since no tests are executed, hence we get zero coverage.

I tried to do simple dotnet test Microsoft.Extensions.Compliance.Abstractions.Tests.dll, and I received this.

Test run for D:\Issues\Net462\extensions\artifacts\bin\Microsoft.Extensions.Compliance.Abstractions.Tests\Debug\net462\Microsoft.Extensions.Compliance.Abstractions.Tests.dll (.NETFramework,Version=v4.6.2)
Microsoft (R) Test Execution Command Line Tool Version 17.7.0-preview-23317-01+919ec8358820228cc5fa77ef000051c1d6875399 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:04.98] Skipping: Microsoft.Extensions.Compliance.Abstractions.Tests (could not find dependent assembly 'Microsoft.Extensions.Compliance.Abstractions.Tests, Version=8.0.0')
No test is available in D:\Issues\Net462\extensions\artifacts\bin\Microsoft.Extensions.Compliance.Abstractions.Tests\Debug\net462\Microsoft.Extensions.Compliance.Abstractions.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

[jakubch1]

@RussKie @geeknoid I think you should add option to fail if no tests executed: <TreatNoTestsAsError>true</TreatNoTestsAsError>
microsoft/vstest#2247