Does it make any difference if the projects being tested/covered are built in DEBUG or RELEASE mode?

[PureKrome]

Does it matter if the projects which are getting tested against, if they are built using DEBUG or RELEASE configuration?

I know the test project isn't covered so I don't care what that is built against ... but the test project tests OTHER netstandard libraries / netcore apps ... which I'm wondering if I do care about the 'configuration' setting for those.

[MarcoRossignoli]

Yep the configuration changes the results because compiler could emit different IL so you can run you CI with both configuration(I advice to do that for testing reason) but you could see slightly different coverage result, check this old issue for instance #389

[MarcoRossignoli]

For testing perspective it's very different usually related to performance always from emitting side and also because usually in debug/checked mode you run assertion and preprocessor use #define DEBUG or [Conditional("DEBUG")] for istance diagnostics assertion https://github.com/dotnet/runtime/blob/4f9ae42d861fcb4be2fcd5d3d55d5f227d30e723/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs#L88

Semantics must remain the same.

[PureKrome]

Thanks heaps @MarcoRossignoli for the prompt reply 👍

I sorta don't understand what you are saying though 😊

Yes -> DEBUG creates different IL compared to RELEASE. Okay, that make sense .. but what should I be using for code coverage?

I had a read over #389 and it was a bit .. technical for me 😊

So even though there's different IL, should we be testing our projects that are compiled under DEBUG configuration for code coverage because there's less optimizations, etc?

[MarcoRossignoli]

DO testing DEBUG/RELEASE and check coverage report under DEBUG, btw if you've a simmetric CI where you simply pass $(configuration) as parameter won't be a problem you'll test/cover both config and it's ok. Debug report of coverage are usually more "precise" related to covered line because IL/pdb emitted by compiler is more "detailed".

[MarcoRossignoli]

For instance check coverlet CI https://github.com/tonerdo/coverlet/blob/master/eng/build.yml we build test and cover in both configuration...I'll check only debug one.

[MarcoRossignoli]

I mean usually debug it's enough, but suppose to have a bug where some code run only in debug mode and with that coverage percentage reach 90% but you'll find that in release mode drops under 80%...likely it's something to double check.

[PureKrome]

Oh wow! that's ... really interesting.

CI/CD Summary

• Run all builds and tests in DEBUG and RELEASE configurations (leveraging the matrix ability for this!)
• Only publish the RELEASE nugets.

But how is is the code coverage kicked off?
Checking the latest build (at the time of me posting this reply) at Azure Dev Ops shows this:

yml task:

- task: DotNetCoreCLI@2
  displayName: Test
  inputs:
    command: test
    arguments: -c $(BuildConfiguration) --no-build /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:Include=[coverlet.*]* /p:Exclude=[coverlet.tests.remoteexecutor]*
    testRunTitle: $(Agent.JobName)

build log:

[command]C:\hostedtoolcache\windows\dotnet\dotnet.exe test --logger trx --results-directory d:\a\_temp -c Debug --no-build /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:Include=[coverlet.*]* /p:Exclude=[coverlet.tests.remoteexecutor]*
Test run for d:\a\1\s\test\coverlet.core.tests\bin\Debug\netcoreapp2.2\coverlet.core.tests.dll(.NETCoreApp,Version=v2.2)

which suggests it's the MSBuild way? Where is the msbuild stuff defined/set? I also thought that the preferred way is to do this via the coverlet.collector nuget package, not msbuild? (I personally prefer dotnet tool's over msbuild )

[MarcoRossignoli]

Yes we use msbuild at the moment because we know that it doesn't affected by know issue, so you should use collectors if possible.

[MarcoRossignoli]

For us it's also a way to test engine, we use code just build and we inject it through msbuild targets, but it's only for internal use.

[PureKrome]

Okay, so if I use the coverlet.collector by referencing the nuget package in my test project, will that ignore that test project by default ?

[MarcoRossignoli]

Yep, you can follow our hello world sample https://github.com/tonerdo/coverlet/tree/master/Documentation/Examples/VSTest/HelloWorld with and without runsettings the internal engine it's the same, so also the options we call msbuild/vstest/.net tools drivers that run same engine, the only difference is that some "config options" could be not "exposed" for some drivers, but we can add i needed.

[PureKrome]

Hi again @MarcoRossignoli - I'm back here trying to get this working once more.

TL;DR;

I'm not getting any summary of results, listed to the terminal.

Info:

So i'm trying to learn off your repo azure-pipelines.yml file. In it you have various OS flavours and are doing both DEBUG and RELEASE builds. Great.

In all of the TEST step, it produces a nice graphical result to the logging console out.
e.g.

Calculating coverage result...
  Generating report '/home/vsts/work/1/s/test/coverlet.core.tests/coverage.opencover.xml'

+------------------------------------+--------+--------+--------+
| Module                             | Line   | Branch | Method |
+------------------------------------+--------+--------+--------+
| coverlet.tests.projectsample.empty | 0%     | 100%   | 0%     |
+------------------------------------+--------+--------+--------+
| coverlet.msbuild.tasks             | 0%     | 0%     | 0%     |
+------------------------------------+--------+--------+--------+
| coverlet.core                      | 83.65% | 78.2%  | 84.07% |
+------------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 71.74% | 69.83% | 61.9%  |
+---------+--------+--------+--------+
| Average | 27.88% | 59.4%  | 28.02% |
+---------+--------+--------+--------+

I'm not getting this report into my logs. I'm using the "collector" solution (not msbuild, not global tool). Is that why?

e.g.

steps:
- task: DotNetCoreCLI@2
  displayName: 'Tests' 
  inputs:
    command: test
    projects: '${{ parameters.solutionFile }}'
    arguments: -c $(BuildConfiguration) --no-build --collect:"XPlat Code Coverage"
    testRunTitle: $(Agent.JobName)

and each xunit test project includes the nuget package...

<ItemGroup>
  <PackageReference Include="coverlet.collector" Version="1.2.0">
    <PrivateAssets>all</PrivateAssets>
    <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
  </PackageReference>
  
  ...<skip>...

</ItemGroup>

When I look at the logs in AzureDevOps, I can see this at the top ...

/usr/bin/dotnet test /home/vsts/work/1/s/Foo.Search.sln --logger trx --results-directory /home/vsts/work/_temp -c Debug --no-build --collect:XPlat Code Coverage

so it looks for all the tests in the .sln and finds them and runs the discovered tests. Here's an example output:

Results File: /home/vsts/work/_temp/_fv-az175_2020-01-08_07_11_45.trx

Attachments:
  /home/vsts/work/_temp/58960cd6-99b6-47b1-8468-6df3845c9ed7/coverage.cobertura.xml
Test Run Successful.
Total tests: 52
     Passed: 52
 Total time: 26.6901 Seconds

So it looks like:

• it created a tests results file (that .trx file)
• some coverage file (that coverage.cobertura.xml file)

but noticed how it says: Attachments .. is that causing this to not print? This repo ReadMe.md says:

A summary of the results will also be displayed in the terminal.

which is not occurring :(

I also tried using change the format to opencover, but that errored all the time => wrong or bad switch. e.g. arguments: -c $(BuildConfiguration) --no-build --collect:"XPlat Code Coverage" --format:opencover [there's no docs explaining what the syntax is]

[MarcoRossignoli]

Simply printing results to console is not supported for collectors for the moment.

Collectors are a totally different beast than .net tool and msbuild, we don't have complete control of workflow but we're injected inside the vstest engine.
BTW I'll open a new issue to understand if/how we can show reports on console like .net tools and msbuild.

Opened a issue #681

[PureKrome]

Kewl thanks - much appreciated.

Q1: So even though there's no output to the terminal (reported separately in #681) it still creates the coverage xml file ... which means we could still use a 3rd party tool to help interegate the coverage results (e.g. codecov.io or ReportGenerator), right?

Q2: Secondly, in your azure pipelines for this repo, you so DEBUG and RELEASE. How do you only get coverage for DEBUG tests? It feels like each OS generates coverage for DEBUG and RELEASE mode? The scenario is that I wish to generate coverage for DEBUG only (as suggested by you, above) and then use a 3rd party to interigate the results.

Q3: Lastly, how do you specify the format using the coverage tool. The collector docs say the only optional parameter is format but I can't figure out how to use that param properly, on the command line.

[MarcoRossignoli]

Q1: So even though there's no output to the terminal (reported separately in #681) it still creates the coverage xml file ... which means we could still use a 3rd party tool to help interegate the coverage results (e.g. codecov.io or ReportGenerator), right?

Yes

Q2: Secondly, in your azure pipelines for this repo, you so DEBUG and RELEASE. How do you only get coverage for DEBUG tests? It feels like each OS generates coverage for DEBUG and RELEASE mode? The scenario is that I wish to generate coverage for DEBUG only (as suggested by you, above) and then use a 3rd party to interigate the results.

If you want keep coverage for release but they are different because different IL is emitted by compiler.

Q3: Lastly, how do you specify the format using the coverage tool. The collector docs say the only optional parameter is format but I can't figure out how to use that param properly, on the command line.

You need to provide a runsetting file and inside format element include the list of formats https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md#advanced-options-supported-via-runsettings
Take a look to samples https://github.com/coverlet-coverage/coverlet/tree/master/Documentation/Examples/VSTest/HelloWorld

I apologize for late answer I lost this last comment and I don't know why 🙇
Feel free to close if solved!

[PureKrome]

No probs at all.

Since I got some answers and clues to this issue, a few months ago, I've started doing coverlet coverage for DEBUG builds. I'm also doing a "matrix" which does both DEBUG and RELEASE builds at the same time:

• coverlet for DEBUG
• create publish artifacts for RELEASE (aka pack)

to do this, i'm using a powershell script which does one or the other, based on the 'configuration' (D or R).

works great :)

cheers 👍

[MarcoRossignoli]

Glad to hear!feel free to ask if you're in trouble!