Coverage is not calculated if there is rabbitmq used

[fkucuk]

First of all, thanks a lot for this great tool.

I am testing my REST Api by following Microsofts this guide: https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-2.2

In this guide, I am extending dotnet.core's WebApplicationFactory:

public class TestWebApplicationFactory<TStartup> : WebApplicationFactory<TStartup> where TStartup : class

My sample repo is here: https://github.com/fkucuk/shared/tree/master/CoverletCoverageIssue

In the startup of my application I configure RabbitMQ. On the test project I can override this configuration. If I don't, its own configuration is used.

The interesting thing here is, while testing if I don't override the setting in the Startup.cs the coverage report is as follows.

In this execution, dotnet uses the appsettings.Development.json file on the api project, and connects the real rabbitmq server.

But if I override the rabbit config and use fake of the rabbit, in memory, coverage works perfectly fine:

Accessing an outer environment, interestingly, interrupts calculation of coverage.

I know, when testing I have to fake or mock real systems. But I am scared if any other behavior may cause coverlet coverage calculation to fail.

What I am trying to ask here is, is this the expected behavior?

[MarcoRossignoli]

I cannot repro, I mean if I start test startup code run and I don't have correct rabbit configuration and get exceptions. How you run in "mock" mode?

Copyright (c) Microsoft Corporation. Tutti i diritti sono riservati.

Avvio dell'esecuzione dei test in corso. Attendere...
[xUnit.net 00:00:01.82]     ApiTests.UnitTest1.Test2 [FAIL]
[xUnit.net 00:00:01.83]     ApiTests.UnitTest1.Test1 [FAIL]
  X ApiTests.UnitTest1.Test2 [223ms]
  Messaggio di errore:
   System.InvalidOperationException : Failed to convert configuration value to type 'System.Int32'.
---- System.ArgumentException : portNo is not a valid value for Int32.
Parameter name: value
-------- System.FormatException : Input string was not in a correct format.
  Analisi dello stack:
     at Microsoft.Extensions.Configuration.ConfigurationBinder.BindInstance(Type type, Object instance, IConfiguration config, BinderOptions options)
   at Microsoft.Extensions.Configuration.ConfigurationBinder.BindProperty(PropertyInfo property, Object instance, IConfiguration config, BinderOptions options)
   at Microsoft.Extensions.Configuration.ConfigurationBinder.BindNonScalar(IConfiguration configuration, Object instance, BinderOptions options)
   at Microsoft.Extensions.Configuration.ConfigurationBinder.BindInstance(Type type, Object instance, IConfiguration config, BinderOptions options)
   at Microsoft.Extensions.Configuration.ConfigurationBinder.Bind(IConfiguration configuration, Object instance, Action`1 configureOptions)
   at Microsoft.Extensions.Configuration.ConfigurationBinder.Bind(IConfiguration configuration, Object instance)
   at Microsoft.Extensions.Configuration.ConfigurationBinder.Bind(IConfiguration configuration, String key, Object instance)
   at CoverletIssueRaiser.Startup.ConfigureServices(IServiceCollection services) in C:\git\shared\CoverletCoverageIssue\CoverletIssueRaiser\Startup.cs:line 32
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.Initialize()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at Microsoft.AspNetCore.TestHost.TestServer..ctor(IWebHostBuilder builder, IFeatureCollection featureCollection)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateServer(IWebHostBuilder builder)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.EnsureServer()
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(DelegatingHandler[] handlers)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateClient(WebApplicationFactoryClientOptions options)
   at ApiTests.UnitTest1.Test2() in C:\git\shared\CoverletCoverageIssue\ApiTests\UnitTest1.cs:line 29
--- End of stack trace from previous location where exception was thrown ---
----- Inner Stack Trace -----
   at System.ComponentModel.BaseNumberConverter.ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, Object value)
   at System.ComponentModel.TypeConverter.ConvertFromInvariantString(String text)
   at Microsoft.Extensions.Configuration.ConfigurationBinder.TryConvertValue(Type type, String value, Object& result, Exception& error)
----- Inner Stack Trace -----
   at System.Number.StringToNumber(ReadOnlySpan`1 str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   at System.Number.ParseInt32(ReadOnlySpan`1 s, NumberStyles style, NumberFormatInfo info)
   at System.Int32.Parse(String s, NumberStyles style, IFormatProvider provider)
   at System.ComponentModel.Int32Converter.FromString(String value, NumberFormatInfo formatInfo)
   at System.ComponentModel.BaseNumberConverter.ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, Object value)
  X ApiTests.UnitTest1.Test1 [18ms]
  Messaggio di errore:

Can you also try to enable logging to undestand what is instrumented? https://github.com/tonerdo/coverlet/blob/master/Documentation/Troubleshooting.md#msbuild-integration

[fkucuk]

hey @MarcoRossignoli thanks for trying. You need a real rabbit server and change the configuration file properly.

I've masked my settings, so you get error.

[MarcoRossignoli]

Ok so your tests run mocked by WebApplicationFactory but only if there is a rabbit server?
I mean you should try to not use real rabbit if isRabbitFake, is it possible?
BTW have you tried to enable logs to understand if dll is instrumented?
0% could be also related to a know issue #110
Can you try with console tool(no msbuild) https://github.com/tonerdo/coverlet/blob/master/Documentation/GlobalTool.md?

[fkucuk]

Sorry for providing a nonworking sample. But you need a working rabbit to run my sample.

I can fake/mock rabbit server by using its InMemory feature, with the following code:

public class TestWebApplicationFactory<TStartup> : WebApplicationFactory<TStartup> where TStartup : class
    {
        protected override void ConfigureWebHost(IWebHostBuilder builder)
        {
            builder.ConfigureTestServices(services =>
            {
                services.AddSingleton(provider => Bus.Factory.CreateUsingInMemory(configure =>
                {
                    configure.ReceiveEndpoint("testendpoint", endpoint =>
                    {
                        endpoint.Consumer<MessageConsumer>(provider);
                        EndpointConvention.Map<SendMessageEvent>(endpoint.InputAddress);
                    });
                }));
                services.AddSingleton<IHostedService, MessageMassTransitService>();
            });

        }
    }

And it works perfectly fine. I mean, the coverage is calculated.

But If I don't use fake/mock of the rabbit. I mean, if I access the real rabbitmq server from the running test environment, the coverage results are not calculated. With the conditional statement in my test, I have tried to demonstrate the difference.

If the condition is true, my code uses inmemory rabbit, and coverage is calculated.
If the condition is false, my code uses the configuration of the main program (appsettings.Development.json) and the code in the Startup.cs file is used, and the coverage is not calculated. And yes, you must provide a real rabbit server's configuration to run this application.

[fkucuk]

I have tried with the global tool, unfortunately, the result is the same.

I've also run the tests with the following command:
dotnet test -c debug /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:Include=[coverlet.*]* -verbosity:diagnostic -bl:realmsbuild.binlog -noconsolelogger for two versions of my code.

I've set isRabbitFake = true

The timeline window:

But when I set it to false

Interesting thing here is, "GenerateCoverageResult" step is missing with the tests running on real RabbitServer,

[MarcoRossignoli]

Dup of #573 (comment)