nunit3-console with altcover 0% coverage

[39547361]

For some strange reason i get 0% coverage with altocover (monocov reports 90%). Am i missing something? Any hints would be greatly appreciated.

altcover.5.0.663
MacOSX 17.7.0.0
CLR Version: 4.0.30319.42000

Step 1: create pcl and nunit test dll

nuget restore
/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild TESTReport.sln /t:TESTReportPCL /p:Configuration="Debug" /p:BuildProjectReferences=false /m  /verbosity:m
/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild TESTReport.sln /t:NUnit_TESTReport /p:BuildProjectReferences=false /m  /verbosity:m

Step 2

cd NUnit.TESTReport/bin/Debug/

Step 3

mono ../../../packages/altcover.5.0.663/tools/net45/AltCover.exe --opencover --linecover  --assemblyFilter=.+.TESTReport+ --assemblyFilter=TESTReport+ --assemblyExcludeFilter=nunit+ --assemblyExcludeFilter=.+\.Tests --assemblyExcludeFilter=NUnit+ --assemblyExcludeFilter=AltCover.+ --assemblyExcludeFilter=Xamarin+ --assemblyExcludeFilter=SQL+ --assemblyExcludeFilter=System+ --assemblyExcludeFilter=Plugin+ --assemblyExcludeFilter=PCLExt+ --assemblyExcludeFilter=FFImage+ --assemblyExcludeFilter=Newtonsoft+  --outputDirectory=__Instrumented

Output:

Instrumenting files from /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug
Writing files to /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/TestReportPCL.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/FFImageLoading.Forms.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/FFImageLoading.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/NUnit.TestReport.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/Newtonsoft.Json.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/Plugin.Connectivity.Abstractions.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/Plugin.Connectivity.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/SQLite-net.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/SQLiteNetExtensions.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/Xamarin.Forms.Core.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/Xamarin.Forms.Xaml.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/nunit.framework.dll

Coverage Report: /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/coverage.xml


    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/nunit.framework.dll
                <=  nunit.framework, Version=3.11.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/Xamarin.Forms.Core.dll
                <=  Xamarin.Forms.Core, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/SQLite-net.dll
                <=  SQLite-net, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/Plugin.Connectivity.Abstractions.dll
                <=  Plugin.Connectivity.Abstractions, Version=3.2.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/Newtonsoft.Json.dll
                <=  Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/FFImageLoading.dll
                <=  FFImageLoading, Version=2.4.4.859, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/Xamarin.Forms.Xaml.dll
                <=  Xamarin.Forms.Xaml, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/SQLiteNetExtensions.dll
                <=  SQLiteNetExtensions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/Plugin.Connectivity.dll
                <=  Plugin.Connectivity, Version=3.2.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/FFImageLoading.Forms.dll
                <=  FFImageLoading.Forms, Version=2.4.4.859, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/TestReportPCL.dll
                <=  TestReportPCL, Version=1.0.6941.28281, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/NUnit.TestReport.dll
                <=  NUnit.TestReport, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null

Generated files:

AltCover.Recorder.g.dll
TestReportPCL.dll
TestReportPCL.dll.config
TestReportPCL.pdb
CarouselView.FormsPlugin.Abstractions.dll
coverage.xml
FFImageLoading.dll
FFImageLoading.Forms.dll
FFImageLoading.Forms.pdb
FFImageLoading.Forms.xml
FFImageLoading.pdb
FFImageLoading.Platform.dll
FFImageLoading.Platform.xml
FFImageLoading.Transformations.dll
FFImageLoading.Transformations.xml
FFImageLoading.xml
I18NPortable.dll
libe_sqlite3.dylib
Newtonsoft.Json.dll
Newtonsoft.Json.pdb
Newtonsoft.Json.xml
NUnit.TestReport.dll
NUnit.TestReport.dll.config
NUnit.TestReport.pdb
nunit.framework.dll
nunit.framework.pdb
nunit.framework.xml
PCLExt.FileStorage.Abstractions.dll
PCLExt.FileStorage.Abstractions.xml
PCLExt.FileStorage.dll
PCLExt.FileStorage.xml
Plugin.Connectivity.Abstractions.dll
Plugin.Connectivity.Abstractions.pdb
Plugin.Connectivity.Abstractions.xml
Plugin.Connectivity.dll
Plugin.Connectivity.pdb
Plugin.Connectivity.xml
Plugin.Media.dll
Plugin.Media.xml
Plugin.Permissions.dll
Plugin.Permissions.xml
SignaturePad.Forms.dll
SignaturePad.Forms.xml
SQLite-net.dll
SQLite-net.pdb
SQLite-net.xml
SQLiteNetExtensions.dll
SQLiteNetExtensions.pdb
SQLitePCLRaw.batteries_green.dll
SQLitePCLRaw.batteries_v2.dll
SQLitePCLRaw.core.dll
SQLitePCLRaw.provider.e_sqlite3.dll
System.Data.Common.dll
System.Diagnostics.StackTrace.dll
System.Diagnostics.Tracing.dll
System.Globalization.Extensions.dll
System.IO.Compression.dll
System.IO.dll
System.Linq.dll
System.Linq.Expressions.dll
System.Net.Http.dll
System.Net.Sockets.dll
System.Reflection.dll
System.Runtime.dll
System.Runtime.Extensions.dll
System.Runtime.InteropServices.dll
System.Runtime.InteropServices.RuntimeInformation.dll
System.Runtime.Serialization.Primitives.dll
System.Security.Cryptography.Algorithms.dll
System.Security.Cryptography.Encoding.dll
System.Security.Cryptography.Primitives.dll
System.Security.Cryptography.X509Certificates.dll
System.Security.SecureString.dll
System.Text.RegularExpressions.dll
System.Threading.Overlapped.dll
System.Xml.ReaderWriter.dll
System.Xml.XPath.XDocument.dll
Xamarin.Forms.Core.dll
Xamarin.Forms.Core.pdb
Xamarin.Forms.Core.xml
Xamarin.Forms.Platform.dll
Xamarin.Forms.Xaml.dll
Xamarin.Forms.Xaml.pdb
Xamarin.Forms.Xaml.xml

coverage xml looks fine

Step 4:

mono ../../../packages/altcover.5.0.663/tools/net45/AltCover.exe Runner --recorderDirectory __Instrumented  --executable ../../../packages/NUnit.ConsoleRunner.3.9.0/tools/nunit3-console.exe   -- ../../../NUnit.TESTReport/bin/Debug/NUnit.TESTReport.dll

Output:

Beginning run...
Command line : '../../../packages/NUnit.ConsoleRunner.3.9.0/tools/nunit3-console.exe ../../../NUnit.TestReport/bin/Debug/NUnit.TestReport.dll'
NUnit Console Runner 3.9.0
Copyright (c) 2018 Charlie Poole, Rob Prouse
Runtime Environment
   OS Version: MacOSX 17.7.0.0 
  CLR Version: 4.0.30319.42000
Test Files
    ../../../NUnit.TestReport/bin/Debug/NUnit.TestReport.dll
Run Settings
    DisposeRunners: True
    WorkDirectory: /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.7.1
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 8
Test Run Summary
  Overall result: Passed
  Test Count: 99, Passed: 99, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-01-03 08:34:47Z
    End time: 2019-01-03 08:34:56Z
    Duration: 9.515 seconds
Results (nunit3) saved as TestResult.xml
Getting results...
A total of 0 visits recorded
Coverage statistics flushing took 0.13 seconds
Visited Classes 0 of 39 (0)
Visited Methods 0 of 71 (0)
Visited Points 0 of 649 (0)
Visited Branches 0 of 71 (0)
==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 0 of 46 (0)
Alternative Visited Methods 0 of 119 (0)

[SteveGilham]

Looks like the coverage results aren't being properly flushed to disk on process termination; the same underlying issue as #42

[39547361]

release/v4.0.659 (that fixed #42) does not fix this issue

[SteveGilham]

Actually, issue #42 was re-opened a couple of weeks before this issue was raised. Getting a repro on that is next on my list when I get a chance to resume work on this project.

[39547361]

Sorry for misleading comment. What i meant to say was:
This issue still present with an older altcover release/v4.0.659
it might be relevant in so far that issue #42 is not occurring with altcover release/v4.0.659

[SteveGilham]

As the only change near the relevant code between 4.0.659 and 5.0.663 was a change in the static linkage of the F# core assembly, the original "fix" may have been more a "working by coincidence".

Given that the whole system relies on any outstanding data being flushed during the process shutdown handler, I can see a possibility where very small or very large programs-under-test work, but moderately sized ones fail with zero coverage, so I need to test that hypothesis.

[SteveGilham]

I've not been able to get any reliable repro, but I have done some tuning on the recording side in release 5.0.665, which may ameliorate this problem.

[39547361]

Thank you so far for your effort. Unfortunately 5.0.665 does not fix this particular issue.
I created a private sample project and sent you an invite in case you are interested.

[SteveGilham]

I was afraid that I wouldn't have fixed it; I have a very intermittent case which showed up for the first time in a while after I'd made the release.

I'm not seeing your invite anywhere obvious from my side, though.

[SteveGilham]

The invite appeared a mailbox sub-folder, rather than anywhere on my GitHub profile. I hadn't been expecting that.

[SteveGilham]

Thank you for that. It reproduces on Windows.

[SteveGilham]

The first thing I noticed in looking at the coverage.xml file was that every assembly was marked skippedDueTo="Filter"; the second was that the module under test, AltcoverTestPCL wasn't even present there.

I proceeded to investigate by the following steps

1. recursively delete NUnit.AltcoverTest/bin/Debug/
2. rebuild NUnit.AltcoverTest.csproj
3. cd NUnit.AltcoverTest/bin/Debug/
4. run AltCover.exe --opencover --linecover --assemblyExcludeFilter="^((?!AltcoverTestPCL).)*$" --outputDirectory=__Instrumented using a regex to simply exclude every assembly not containing AltcoverTestPCL in its name
5. as this is Windows, manually copy the x86 and x64 folders to __Instrumented (a limitation; I could have used --inplace in step 4 instead); other platforms will just place libe_sqlite3.so in the Debug folder where it will get copied as expected
6. run AltCover.exe Runner --recorderDirectory __Instrumented --executable ../../../packages/NUnit.ConsoleRunner.3.9.0/tools/nunit3-console.exe --./__Instrumented/NUnit.AltcoverTest.dll to point NUnit at the instrumented test system in __Instrumented.
7. See the following

11,931 visits recorded in 00:00:00.2904547 (41,077 visits/sec)
A total of 11,931 visits recorded
Coverage statistics flushing took 0.38 seconds
Visited Classes 43 of 43 (100)
Visited Methods 83 of 84 (98.81)
Visited Points 543 of 582 (93.3)
Visited Branches 0 of 84 (0)
==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 43 of 54 (79.63)
Alternative Visited Methods 83 of 132 (62.88)

[39547361]

Thank you very much for resolving my misconfiguration. Have a great day sir!

[SteveGilham]

FWIW, the issue in point 5 of my comment of the 8th Feb 2019 is belatedly resolved -- the manual copy of the x86 and x64 folders is no longer needed, as this is done automatically since today's release of v8.2.825; at the time I was too busy focussing on the zero coverage issue to heed my own words about this being a limitation.