Converted solution to VS2017 format and dotnet test, and now missing coverage

[snakefoot]

My Framework

• .NET 4.5
• .NET 4.6.1

My Environment

• AppVeyor (VS2017 Image - OpenCover.4.6.519)

I have already...

• repeated the problem using the latest stable release of OpenCover.
• reviewed the usage guide and usage document.
• have looked at the opencover output xml file in an attempt to resolve the issue.

My issue is related to (check only those which apply):

• almost no coverage being recorded

Expected Behavior

Should include coverage of entire library-code.

Actual Behavior

When using filter, then it only include a single class (BaseMutexFileAppender.cs)

Without filter then it shows the correct coverage of the unit-test-library, but still only a single class of the library being tested.

Steps to reproduce the problem:

• Execute OpenCover with dotnet test on AppVeyor VS2017-Image:

OpenCover.Console.exe -register:user -target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:"test tests\NLog.UnitTests --configuration Debug --framework net461" -returntargetcode -filter:"+[NLog]* +[NLog.Extended]* -[NLog]JetBrains.Annotations.*" -excludebyattribute:*.ExcludeFromCodeCoverage* -hideskipped:All -output:coverage.xml

• Broken coverage result - Notice how it only shows for BaseMutexFileAppender.cs
• Broken coverage build report
• appveyor.yml
• Library csproj
• UnitTest csproj

[sawilde]

You'll probably need to try that latest beta build to get coverage from anything built using dotnet this is due to changes in pdb formats that are not supported by the current release.

https://ci.appveyor.com/project/sawilde/opencover/build/4.6.780/artifacts

[snakefoot]

@sawilde Funny how the UnitTest-classes gets coverage, but not the actually Library-classes.

Possible for you to publish a release-candidate nuget-package, that I can grap from the NuGet-feed (easier to use on appveyor.yml) ?

[sawilde]

@snakefoot not yet as it isn't ready - publishing a release candidate creates an expectation

[sawilde]

There is a myget feed - https://www.myget.org/feed/Packages/opencover

[snakefoot]

@sawilde Is this a public feed ? I cannot seem to access it.

[sawilde]

@snakefoot I believe so but you may need a myget account perhaps.

try this:

nuget.exe install OpenCover -Version 4.6.780-rc -Source https://www.myget.org/F/opencover/api/v3/index.json

[snakefoot]

@sawilde Tried the BETA-build, but it is also only picking up some of the coverage:

https://codecov.io/gh/NLog/NLog/commit/615e6d61353a230968c87d5c4dc3641a7f8ca29c/build

This is the normal test-complete output:

=== TEST EXECUTION SUMMARY ===
   NLog.UnitTests  Total: 2225, Errors: 0, Failed: 0, Skipped: 6, Time: 310.009s
Committing...
Visited Classes 390 of 419 (93.08)
Visited Methods 3233 of 3694 (87.52)
Visited Points 19184 of 22782 (84.21)
Visited Branches 7360 of 9066 (81.18)

This is the output with the new solution:

Total tests: 2223. Passed: 2217. Failed: 0. Skipped: 6.
Test Run Successful.
Test execution time: 5.1928 Minutes
Committing...
Visited Classes 130 of 418 (31.1)
Visited Methods 835 of 3695 (22.6)
Visited Points 4977 of 22797 (21.83)
Visited Branches 1871 of 9932 (18.84)

[snakefoot]

Discovered that xUnit 2.0 xunit.console.x86.exe is still able to execute the tests, and now it the result is this:

=== TEST EXECUTION SUMMARY ===
   NLog.UnitTests  Total: 2223, Errors: 0, Failed: 0, Skipped: 6, Time: 412,726s
Committing...
Visited Classes 390 of 418 (93.3)
Visited Methods 3241 of 3695 (87.71)
Visited Points 19233 of 22821 (84.28)
Visited Branches 7379 of 9068 (81.37)

[304NotModified]

Have the same problem here (NLog/NLog.MailKit#5), and that's a lot smaller repo (so easier to test)

[quetzalcoatl]

I've noticed @snakefoot was effectively using dotnet.exe test ... --framework net461 as the OpenCover target command. I have just succeeded in running similar thing, and this seems to be a good place to quickly write some details for other people who try to get it running on various target platforms.

My story is - I work on Win10-Home x64, VS2017 Comunity, OpenCover 4.6.519, xunit mixed 2.2.0 and 2.3.1, and I have a netcoreapp2.0 projects that I wanted to cover. I succeeded, and although that's offtopic another story I've yet to blog about, let me summarize the most important parts for comparison:

(netcoreapp2.0) In covered project csproj:

<TargetFramework>netcoreapp2.0</TargetFramework>
<DebugType>full</DebugType>

(netcoreapp2.0) In test project:

<TargetFramework>netcoreapp2.0</TargetFramework>
<!-- <DebugType>full</DebugType> - NOPE! -->

(netcoreapp2.0) OpenCover runner (single command line, multiline for readability):

OpenCover.Console.exe
    -register:user
    -target:"C:\Program Files\dotnet\dotnet.exe"
    -targetargs:"test --framework netcoreapp2.0 --logger:trx \"MyProject\MyProject.csproj\" "
    -output:"MyProject\TestResults\results-coverage.xml"
    -oldstyle
    -log:Warn

I've spent few days (really, not exagerrating) on researching different options, and that's the only combination that worked. I excluded unimportant things like coverbytest, filters, etc - there were no issues with them, these worked as expected. Similar setup worked for me for netcoreapp1.1 and netcoreapp2.0. Note that I had to enable Full PDBs only for "real projects", and enabling them for test projects did some damage, although I don't remember what exactly (sorry, that was over 2 months ago). Another thing to note is the -oldstyle switch already discussed in other issues. Warn or Info log level helps you to see "No PDBs" infos which help in diagnosing when you get empty reports. I also had to set up output and logger due to multiple projects and multiple test projects to cover in one go.

Now the fun part. I had to switch my net-core-app to net461 due to 3rd party libraries. Application works fine, tests run fine, opencover (--framework net461) returns:

(log level: all)
Cannot instrument [SNIP]\AppData\Local\Temp\[SNIP]\assembly\dl3\[SNIP]\Foobar.WebApi.exe as no PDB/MDB could be loaded

(log level: warn)
No results, this could be for a number of reasons. The most common reasons are:
    1) missing PDBs for the assemblies that match the filter please review the
    output file and refer to the Usage guide (Usage.rtf) about filters.
    2) the profiler may not be registered correctly, please refer to the Usage
    guide and the -register switch.

I similar messages when I battled with netcoreapp, resolved by using full PDBs. Now obviously it was still set, as I just switched the target framework. After another long research, I found out that ... it actually can not locate the PDBs for some reason. Adding -searchdirs parameter made it work for net461!

(net461) In covered project csproj:

<TargetFramework>net461</TargetFramework>
<DebugType>full</DebugType> <!-- dunno, I just left it enabled -->

(net461) In test project:

<TargetFramework>net461</TargetFramework>
<!-- <DebugType>full</DebugType> - dunno, I just left it disabled -->

(net461) OpenCover runner (single command line, multiline for readability, I've left only the important parts):

OpenCover.Console.exe
    -register:user
    -target:"C:\Program Files\dotnet\dotnet.exe"
    -targetargs:"test --framework net461 --logger:trx \"MyProject\MyProject.csproj\" "
    -output:"MyProject\TestResults\results-coverage.xml"
    -oldstyle
    -log:Warn
    -searchdirs:"MyProject\bin\Debug\net461"

I can only guess that either opencover does not look in \bin$config$platform for them, or dotnet test/vstest does not copy them properly, or xunit-runner does not copy/shadow them properly..

By the way, I noticed it still works when I remove, oldstyle. As expected, it is no longer needed as runtime is 461 and new instrumentation can be used. However, for my projects, I left the oldstyle and also left the full PDBs, maybe I'll be switching back to netcore at some point.

Anyways I'm absolutely hyped. For the 3rd time in a row (core 1.1, core 2.0, net461), I was dead-tired trying to get it working and almost gave up, and yet again it works out of the box with OpenCover 4.6.519 provided you somehow happen to guess the switches.

Unfortunatelly, I cannot share the full project/script/msbuild setup, and can't prepare a new self-contained project demos (time/effort/nda/license/etc), I'll try to do that a bit later this or next week.

[sawilde]

@quetzalcoatl thanks - I really must get to updating the documentation around this