Different code coverage results with different sonar scanner version

[ksandrmatveyev]

Description

Hi. I used OpenCover 4.6.519 and Fake (F#) for running of Sonar scanner, msbuild and opencover step-by-step. After migration from Sonarqube 5.5 to Sonarqube 6.7.4 I start to use newest version of Sonar Scanner and receive different code coverage results (from ~86% to ~20%).

• I checked the coverage measures and found that amount of unit tests (nunit) are the same (for both 5.5 and 6.7) but different line of codes and uncovered lines and etc.
• In Treemap found that for 6.7.4 multiple objects are in place, which must be excluded by opencover filters ( -[Tests]* and etc).
• OpenCover versions are the same
• Configurations and cli commands are the same (filters, scanning jobs, opencover and scanner parameters. See Repro steps)
• sonar.coverrage.exclusions are not used
• ReportGenerator shows almost the same results for each test coverage results.

Tried to use different versions of Sonar Scanner for MSBuild:

1. Sonarqube 5.5+Sonar scanner for msbuild 2.0 +sonarc# 5.2 + OpenCover 4.6.519= coverage 86%
2. Sonarqube 6.7.4+Sonar scanner for msbuild 3.0.x + sonarc# 7.1 (build 5212) + OpenCover 4.6.519= coverage 45% (multiple objects were included, but must be excluded)
3. Sonarqube 6.7.4+Sonar scanner for msbuild 4.3.1.1372 (net46) + sonarc# 7.1 (build 5212) + OpenCover 4.6.519= coverage 19% (multiple objects were included, but must be excluded)

Repro steps

D:\tools/MSBuild.SonarQube.Runner/SonarQube.Scanner.MSBuild.exe begin /k:"APP.TEST" /n:"APP.TEST" /v:"1.12-a2d43676" /d:sonar.cs.nunit.reportsPaths=D:\build\output\tests\TestResults.xml /d:sonar.cs.opencover.reportsPaths=D:\build\output\tests\opencoverresults.xml /d:sonar.cpd.cs.minimumLines=3 /d:sonar.cpd.cs.minimumTokens=50 /d:sonar.exclusions=**/*.Designer.cs,**/jquery*.js,**/raphael.js,**/modernizr*.js,**/swfobject*.js,**/waypoints*.js,**/effects.core*.js,**/respond*.js,**/enquire*.js,**/AssemblyInfo.cs,**/AssemblyBusinessLogicInfo.cs,**/GlobalAssemblyInfo.cs,**/AssemblyVersion.cs

C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe  D:\app\app.test.sln /t:Rebuild /m  /nodeReuse:False      /p:RestorePackages="False" /p:Configuration="Release"

D:\tools\OpenCover.Console.exe -target:"D:app\packages\NUnit.ConsoleRunner\tools\nunit3-console.exe" -targetargs:"/config:Release --x86 --result:D:\build\output\tests\TestResults.xml;format=nunit2 D:\app\bin\Release\UnitTests.dll" -output:"D:\build\output\tests\opencoverresults.xml"  -filter:"+[*]* -[*.UI*]* -[*.Tests]* -[*.Testing]* -[*.IntegrationTests]* -[*.Report*]* -[*.Authorization*]* -[HtmlAgilityPack]* -[OpenRasta]* -[RefEmit_*]*" -log:Verbose -register -showunvisited -excludebyattribute:*.ExcludeFromCodeCoverage*

D:\tools/MSBuild.SonarQube.Runner/SonarQube.Scanner.MSBuild.exe end 

Expected behavior

The same code coverage results in sonarqube with different sonar scanners

Actual behavior

Different code coverage results due to including filtered objects

Known workarounds

Missed

Related information

• SonarC# Version - 7.1 (build 5212)
• Scanner for MSBuild Version - 4.3.1.1372 (net46)
• SonarQube version - 6.7.4
• Usage: Standalone/Fake

[ghost]

Just in case (also to avoid any double-investigation), problem also discussed here: https://community.sonarsource.com/t/code-coverage-exclusions-dont-work-for-6-7-x-with-c-projects/1025

[ksandrmatveyev]

Can be closed. Not an issue, see comments there https://community.sonarsource.com/t/code-coverage-exclusions-dont-work-for-6-7-x-with-c-projects/1025/4?u=aleksandr_matveyev