Coverage incorrectly reported as 0% for integration tests

[TheMagnificent11]

I have a open source project that is using Coverlet to calculate test coverage.

The coverage seems to work correctly for unit tests.

However, coverage doesn't get reported for integration tests.

The integration tests I'm running use Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory to bring up a ASP.Net Web API application in memory and then uses a HTTP client to make API calls to this in-memory application.

However, none of the code executed is being included in the coverage. It doesn't even get the top-level controllers that get hit when the HTTP request is executed.

See this pull request for more details.

The dotnet test command is in the GitHub Action (dotnet test ./Lewee-CI.sln --configuration Release --no-build --collect "XPlat Code Coverage" --results-directory ./coverage).

[mklpanasiuk]

Hello @TheMagnificent11. I'm not the developer of Coverlet, but I faced with the same issue while worked with WebApplicationFactory. The root cause in my case of missing coverage results was OneTimeTearDown method (global one), which disposed WebApplicationFactory and their http client. A solution for us was just to remove disposing of them from OneTimeTearDown =)

[TheMagnificent11]

Hello @TheMagnificent11. I'm not the developer of Coverlet, but I faced with the same issue while worked with WebApplicationFactory. The root cause in my case of missing coverage results was OneTimeTearDown method (global one), which disposed WebApplicationFactory and their http client. A solution for us was just to remove disposing of them from OneTimeTearDown =)

I implemented IDisposable on base test class that has a constructor dependency on my implementation of WebApplicationFactory and disposed of it.

Also, each of my tests creates and disposes of the HTTP client. All the tests use the following method to make HTTP requests.

protected async Task<HttpResponseMessage> HttpRequest(HttpMethod method, string apiPath)
{
    using (var request = new HttpRequestMessage(method, apiPath))
    using (var httpClient = this.factory.CreateClient())
    {
        return await httpClient.SendAsync(request);
    }
}

However, I still get 0% coverage for my integration tests 🤷

I'm going to remove my IDisposable implementation has technically the WebApplicationFactory should get disposed by the test runner, which in my case is Xunit.

I wonder if my issue has anything to do with the fact that I'm implementing Xunit.IClassFixture for my base test class.

Are you using Xunit?

[TheMagnificent11]

#901 seems to be related/duplicate of this issue

[TheMagnificent11]

I created a more minimum implementation of integration test using WebApplicationFactory here.

Coverlet correctly calculates coverage, even when I added a await Task.Delay(TimeSpan.FromSeconds(3)) to the API controller to slow it down.

So I'm not sure why the coverage does work in project from my original comment 🤷

[MarcoRossignoli]

Can you try to take a log and verify that all expected dll are instrumented? https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md#collectors-integration

[TheMagnificent11]

@MarcoRossignoli here's my run settings file.

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>cobertura</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
  <InProcDataCollectionRunSettings>
    <InProcDataCollectors>
      <InProcDataCollector
          assemblyQualifiedName="Coverlet.Collector.DataCollection.CoverletInProcDataCollector, coverlet.collector, Version=3.2.0.0, Culture=neutral, PublicKeyToken=null"
          friendlyName="XPlat Code Coverage"
          enabled="True"
          codebase="coverlet.collector.dll" />
    </InProcDataCollectors>
  </InProcDataCollectionRunSettings>
</RunSettings>

And here are the logs.

log.datacollector.23-03-31_21-18-58_26559_1.txt
log.datacollector.23-03-31_21-18-58_40758_1.txt
log.datacollector.23-03-31_21-19-06_23002_1.txt
log.datacollector.23-03-31_21-19-16_04337_1.txt
log.host.23-03-31_21-19-00_72381_10.txt
log.host.23-03-31_21-19-01_36061_10.txt
log.host.23-03-31_21-19-08_13813_10.txt
log.host.23-03-31_21-19-21_76198_7.txt
log.txt

Thanks

[MarcoRossignoli]

You don't need to set the InProcDataCollector pls remove it.

[TheMagnificent11]

@MarcoRossignoli I updated the settings and ran it again.

Here's the updated settings.

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>cobertura</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Here are the new logs.

log.txt
log.host.23-03-31_22-00-10_35130_9.txt
log.host.23-03-31_22-00-11_09346_10.txt
log.host.23-03-31_22-00-17_67089_10.txt
log.host.23-03-31_22-00-31_55099_7.txt
log.datacollector.23-03-31_22-00-08_62380_1.txt
log.datacollector.23-03-31_22-00-15_69739_1.txt
log.datacollector.23-03-31_22-00-25_34881_1.txt
log.datacollector.23-03-31_22-00-07_79695_1.txt

[TheMagnificent11]

@MarcoRossignoli I've created a repository with a more minimal replication of the issue.

The readme explains how to replicate the issue (there is a PowerShell script that runs the appropriate dotnet test command and the generates a coverage report using the Report Generator dotnet tool (it also creates the logs in the ./coverage-logs directory).

My hypothesis is that using an implementation of Xunit.ICollectionFixture<T> to "reset" a database before each test is what causes the issue (there is a very minimal Sqlite database).

Here are the logs I got on my laptop.

log.datacollector.23-04-01_15-39-28_26818_1.txt
log.datacollector.23-04-01_15-39-28_90615_1.txt
log.host.23-04-01_15-39-29_10233_7.txt
log.host.23-04-01_15-39-29_84693_7.txt
log.txt

[TheMagnificent11]

@MarcoRossignoli was that repository that contained more minimal replication of my issue useful?

[MarcoRossignoli]

I didn't have time to check yet, I'm pretty busy this period.

let's see if @daveMueller @petli has got some fresh ideas.

[TheMagnificent11]

I feel rather embarrassed.

The tests that were supposed to contribute to coverage (in both my original repository and the experiment repository) were using BDDfy, but I wasn't adding the BDDfy() method call at the end of my tests 🤦

The super-fast speed of execution of the tests should have been a tell-tale sign.

I made the appropriate changes in my experiment repository and the coverage looked correct.

Sorry about the hassle.