Option --threshold doesn't work, coverage.json is empty

[milica-nikolic]

Hi there!

I am trying to implement Coverlet in my pre-commit script, unit tests are still to be written. I am using this option:
coverlet CBPm.Validations.Test --target "dotnet" --targetargs "test . --no-build" --threshold 80

This snippet sholud retrun non zero value if code coverage is less than 80%.
In my case code coverge is 0% and I still get zero exit code instead of non zero exit code.

In addition to this coverage.json file is generated but it is empty.

Any help would be appreciated.

Kind regards! :)

[milica-nikolic]

I managed to solve problem by using next command:
coverlet CBPm.Validations.Test --target "dotnet" --targetargs "test . --no-build" --threshold 80 --threshold-stat total

So now I'm still wondoring why coverage.json file is empty ? :)

[daveMueller]

I guess your ouput looks somewhat like this?

Calculating coverage result...
  Generating report 'D:\Repos\coverlet\src\coverlet.console\bin\Debug\net5.0\coverage.json'
+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+


D:\Repos\coverlet\src\coverlet.console\bin\Debug\net5.0\coverlet.console.exe (process 25356) exited with code 0.

Your solution doesn't seem to have any module or the module can't be detected by coverlet. Thus coverlet can't instrument any module and isn't able to calculate any coverage. If the coverage result is empty then also the coverage report is empty.

The command in your first comment should return a non zero exit code even when no assemblies can be instrumented. This is a bug, thanks for reporting 👍. It seems that I missed this while working on #1083.

[milica-nikolic]

Thank you for your answer, it really helped me!

I am looking forward for next release :)

And just one additional question, is there any part of documentation that explaines what does procent coverge of branch and method means ?

Thank you!

[daveMueller]

1. Branchpoint that are hit by the tests relative to all branch points of the module.
2. Methods that are hit by the tests relative to the methods count of the module. This doesn't mean that the methods must be completely covered. One covered line in the method counts as hit.

[MarcoRossignoli]

Hi @milica-nikolic,

if you want to try the fix and let us know if it works as expected you can consume the nightly build https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/ConsumeNightlyBuild.md

[milica-nikolic]

Hi again!

Sorry for my late response.

I still have some issues. Running this command:

coverlet Path.To.Tests --target "dotnet" --targetargs "test . --no-build" --threshold 0 --threshold-stat total --format opencover

gives this output

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

In that package I have few tests but i still get zeros in report regadless of those test.

Am I doing something wrong here ?

Additional info I am using this in GitLab CI pipeline as .NET global tool.

[daveMueller]

Like I said before, your solution doesn't seem to have any module that is referenced by your tests or the module can't be detected by coverlet. Usually detection issues are edge cases. Can you provide a repro, so we could analyze this in more detail?

[milica-nikolic]

Hi, thank you for opening this one again.

If by solution you mean .sln file I have all modules included there. I am not able to share my repo unfortunately because it is under company domain.

We have classic arhitecture approach with ui, domain, application, infrastructure and tests.
Test are covering classes and methods from these modules.

[daveMueller]

OK now I have another idea. In the documentation the argument is defined as <ASSEMBLY|DIRECTORY> Path to the test assembly or application directory.

In your example above it says Path.To.Tests. If you have the path to the test source files here it is wrong. It must be the test assembly or the directory where the application assembly is. I think the easiest approach here is to specify the test assembly. Assuming your current directory is the test project:

coverlet ".\bin\Debug\net5.0\TestProject1.dll" --target "dotnet" --targetargs "test . --no-build" --threshold 0 --threshold-stat total --format opencover

If you have multiple test assemblies you have to use merge-with or use reportgenerator to merge them.

[milica-nikolic]

Hi Dave,

Ohh thank you so much for pointing that out, I didn't saw that. After I added path to assembly everything works fine localy, I've got results for coverage. :)

Now I am facing another issue and that is no modules and zero coverage when running in GitLab's pipeline. I have localy .NET 5 installed and coverlet is on version 3.1.0.0 and as I sad everything works great. In my pipeline at first I had this image mcr.microsoft.com/dotnet/nightly/sdk:3.1 and because I could't install coverage I replaced it for mcr.microsoft.com/dotnet/sdk:5.0 and now I have that issue. I have went trough documentation and I found that I need to replace images like that but now I am not sure why I can't see coverage results. Is that one of listed known issues in documentation?

Thank you for provided guidence!

[daveMueller]

OK so your pipeline builds your application inside a docker container? Currently you are using coverlet as a .NET Global Tool. This would require a installation inside the docker image. So every time your docker container starts you would have to install coverlet inside the container before calculating coverage. Or you make a docker file that installs coverlet .NET Global Tool into mcr.microsoft.com/dotnet/sdk and creates a new docker image and you use this new image in your pipeline.

Personally I think it would be much easier to just use coverlet.msbuild or coverlet.collector. There you only have to add a nuget package to your test project(s). Then you just could use the basic sdk:5.0 image provided by MS. Here is the documentation for coverlet.msbuild and for coverlet.collector.

[milica-nikolic]

Hi Dave,

Thank you for your help and instructions.

I managed to get it working using coverlet.msbuild in GitLab CI pipeline, everything works properly now.

Here is snippet of code, maybe it will be helpful for other developers :)

code coverage:
  stage: Test
  image: mcr.microsoft.com/dotnet/nightly/sdk:3.1
  allow_failure: false
  before_script:
    - cd ./Path.To.Tests
    - dotnet add package coverlet.msbuild
  script:
    - cd ..
    - mkdir coverletReport
    - dotnet test /p:CollectCoverage=true /p:Threshold=0 /p:CoverletOutput=testResults/ /p:CoverletOutputFormat=opencover
    - mv ./Path.To.Tests/testResults/coverage.opencover.xml coverletReport
  artifacts:
    when: always
    paths:
      - coverletReport
    expire_in: 8 week
  tags:
    - dev
    - docker