All tests with @retry attributes throwing NullReferenceException - SpecFlow

[ashrafxcoder]

I've started seeing errors with all my integration tests in docker and devops pipelines and the error was only saying NullReference exception, I thought may be something wrong with the dependency injection but that was all working. Only the tests with @retry attributes were failing and after removing the attribute all are passing now.

I got a hint of this problem by googling which took me to this issue: Flickering test: NullReferenceException. Any idea why my tests are failing. I can see @JoshKeegan mentioned that it has been fixed. It may be something else but it is coming from xRetry and I can't see it why.

Exception message:

Error Message:
   System.AggregateException : One or more errors occurred. (Object reference not set to an instance of an object.) (Object reference not set to an instance of an object.)
---- System.NullReferenceException : Object reference not set to an instance of an object.
---- System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace:
  
----- Inner Stack Trace #1 (System.NullReferenceException) -----
   at TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioInitialize(ScenarioInfo scenarioInfo)
   at TechTalk.SpecFlow.TestRunner.OnScenarioInitialize(ScenarioInfo scenarioInfo)
   at IntegrationTests.Features.UserActivityCreatedFeature.ScenarioInitialize(ScenarioInfo scenarioInfo)
   at IntegrationTests.Features.UserActivityCreatedFeature.UserActivityCreated() in/IntegrationTests/Features/UserActivityCreatedFeature.feature:line 6
----- Inner Stack Trace #2 (System.NullReferenceException) -----
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioEnd()
   at TechTalk.SpecFlow.TestRunner.OnScenarioEnd()
   at IntegrationTests.Features.UserActivityCreatedFeature.TestTearDown()
   at IntegrationTests.Features.UserActivityCreatedFeature.System.IDisposable.Dispose()
   at ReflectionAbstractionExtensions.DisposeTestClass(ITest test, Object testClass, IMessageBus messageBus, ExecutionTimer timer, CancellationTokenSource cancellationTokenSource) in /_/src/xunit.execution/Extensions/ReflectionAbstractionExtensions.cs:line 79

My csproj file looks like this:

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.2" />
    <PackageReference Include="SolidToken.SpecFlow.DependencyInjection" Version="3.9.3" />
    <PackageReference Include="SpecFlow" Version="3.9.74" />
    <PackageReference Include="SpecFlow.Plus.LivingDocPlugin" Version="3.9.57" />
    <PackageReference Include="SpecFlow.xUnit" Version="3.9.74" />
    <PackageReference Include="xRetry.SpecFlow" Version="1.8.0" />
    <PackageReference Include="xunit" Version="2.4.2" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.5">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="FluentAssertions" Version="6.8.0" />
  </ItemGroup>

Dependencies registration looks like this:

    public class TestDependencies
    {
        [ScenarioDependencies]
        public static IServiceCollection CreateServices()
        {
            var configuration = new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile("appsettings.json", optional: true)
                .AddEnvironmentVariables()
                .Build();

            var services = new ServiceCollection();
            services.AddLogging();
            services.AddAzureServiceBus(Configuration.ServiceBusConnectionString);

            var options = configuration.GetSection("CosmosDatabase");
            services.AddCosmosRepositories(cosmosDbOptions, "/partitionKey");

            return services;
        }
    }

[JoshKeegan]

Hi @ashrafxcoder,
The issue you linked to was fixed some time ago, but the error stack looks similar so it's likely something else is causing the test to thread hop.

Are you able to reproduce this in a project that you can share so that I can investigate?

[ashrafxcoder]

This is a bit complicated to replicate without the whole test project. As I have mentioned, this is only failing when run in docker. Running outside docker is fine. I will post the solution here if I could replicate it in isolation from our project in docker.