Crash at Microsoft.Extensions.DependencyInjection.ServiceCollectionContainerBuilderExtensions.BuildServiceProvider(IServiceCollection services, ServiceProviderOptions options)

[rhyous]

Description

I am unit testing my DI registrations in dotnet 9.

The unit test usually pass, but every few runs, they fail even though no code has changed. It also doesn't always fail with the same error:

I haven't seen them fail in Debug, only in Release.

See the attached, SuccessfulAndFailedRun.txt.

SuccessfulAndFailedRun.txt

SuccessfulAndFailedRun2.txt

Reproduction Steps

Using the attached project.

Gap.Whitelabel.Microservices.HelloWorld.zip

1. Open the attached project.
2. Run the unit tests.
   I can replicate in VS 2022 by right-clicking and choosing Run Tests
   or in the command line:
   dotnet test --no-build --verbosity normal --filter TestCategory!=Api --collect:"XPlat Code Coverage;Format=cobertura,opencover" --results-directory coverage

Expected behavior

These unit tests should always pass.

Actual behavior

The unit test usually pass, but every few runs, they fail even though no code has changed. It also doesn't always fail with the same error:

Regression?

I reverted to dotnet 8 and tried and got a failure in dotnet 8, too, but the error was again different.

PS C:\dev\ATG\gap-whitelabel-microservice-template\src> dotnet test --no-build --verbosity normal --filter TestCategory!=Api --collect:"XPlat Code Coverage" --results-directory coverage
  Gap.Whitelabel.Microservices.HelloWorld.Api.Tests test succeeded with 1 warning(s) (3.9s)
    C:\Program Files\dotnet\sdk\9.0.103\Microsoft.TestPlatform.targets(48,5): warning : No test matches the given testcase filter `TestCategory!=Api` in C:\dev\ATG\gap-whitelabel-microservice-template\src\Gap.Whitelabel.Microservices.HelloWorld.Api.Tests\bin\Debug\net8.0\Gap.Whitelabel.Microservices.HelloWorld.Api.Tests.dll
Test Parallelization enabled for C:\dev\ATG\gap-whitelabel-microservice-template\src\Gap.Whitelabel.Microservices.HelloWorld.Library.Tests\bin\Debug\net8.0\Gap.Whitelabel.Microservices.HelloWorld.Library.Tests.dll (Workers: 14, Scope: MethodLevel)
Test Parallelization enabled for C:\dev\ATG\gap-whitelabel-microservice-template\src\Gap.Whitelabel.Microservices.HelloWorld.Tests\bin\Debug\net8.0\Gap.Whitelabel.Microservices.HelloWorld.Tests.dll (Workers: 14, Scope: MethodLevel)
  Gap.Whitelabel.Microservices.HelloWorld.Library.Tests test succeeded (5.1s)
  Gap.Whitelabel.Microservices.HelloWorld.Tests test failed with 1 error(s) (5.3s)
    C:\dev\ATG\gap-whitelabel-microservice-template\src\Gap.Whitelabel.Microservices.HelloWorld.Tests\DependencyInjection\HelloWorldModuleTests.cs(75): error TESTERROR:
      HelloWorldModule_Register_IAllStartupValidators_Singleton (311ms): Error Message: Assert.AreSame fail
      ed.
      Stack Trace:
         at Gap.Whitelabel.Microservices.HelloWorld.Tests.DependencyInjection.HelloWorldModuleTests.AssertS
      ingleton[T]() in C:\dev\ATG\gap-whitelabel-microservice-template\src\Gap.Whitelabel.Microservices.Hel
      loWorld.Tests\DependencyInjection\HelloWorldModuleTests.cs:line 75
         at Gap.Whitelabel.Microservices.HelloWorld.Tests.DependencyInjection.HelloWorldModuleTests.HelloWo
      rldModule_Register_IAllStartupValidators_Singleton() in C:\dev\ATG\gap-whitelabel-microservice-templa
      te\src\Gap.Whitelabel.Microservices.HelloWorld.Tests\DependencyInjection\HelloWorldModuleTests.cs:lin
      e 55
         at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean
       isConstructor)
         at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)      

Test summary: total: 14, failed: 1, succeeded: 13, skipped: 0, duration: 5.3s
Build failed with 1 error(s) and 1 warning(s) in 5.9s

Attachments:
  C:\dev\ATG\gap-whitelabel-microservice-template\src\coverage\6cfdffc6-3208-4983-8e31-03403747f05b\coverage.cobertura.xml
  C:\dev\ATG\gap-whitelabel-microservice-template\src\coverage\cb8e727f-970a-44a6-b00b-7ccf25483f3b\coverage.cobertura.xml
  C:\dev\ATG\gap-whitelabel-microservice-template\src\coverage\d9a746e3-0ef5-4b7f-a4ec-b7d8aa43c1dc\coverage.cobertura.xml

Known Workarounds

I haven't found one yet.

Configuration

dotnet 9
Window 11 or the github actions build agents running ubuntu-latests
x64

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-extensions-dependencyinjection
See info in area-owners.md if you want to be subscribed.

[ericstj]

@rhyous why did you make these fields static and why use TestInitialize which runs every method?

        private static IServiceCollection _services;
        private static IServiceProvider _serviceProvider;


        [TestInitialize]
        public void TestInitialize()
        {
            _mockRepository = new MockRepository(MockBehavior.Strict);

            _services = new ServiceCollection();

            // Register upstream registrations (i.e. things that should be pre-registered.)
            _services.AddSingleton<ILoggerFactory, LoggerFactory>();
            _services.AddSingleton(typeof(ILogger<>), typeof(Logger<>));

            // Register this module.
            _services.RegisterModule<HelloWorldModule>();

            var descriptors = _services.ToList();
            var count = descriptors.Count();

            // Build
            _serviceProvider = _services.BuildServiceProvider();
        }

Seems like you could be modifying the _services object on one thread while another is calling ToList(). This will result in the sort of exceptions you were seeing. I made a services local in this method and only assigned to _services at the very end and it resolved the issue for me -- so it does seem like there is a concurrency problem here with your tests.