ObjectDisposedException intermittently crashes test process from DefaultHttpClientFactory.ExpiryTimer_Tick

[martincostello]

Description

I have some repositories that I've hooked up to ingest the daily builds of .NET 8. In this morning's update I have two repos (one public, one private) where the existing test suite is starting to fail due to this error:

The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.<>c.<.cctor>b__22_0(Object s)
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

It appears that something running in the background for HttpClientFactory is trying to use the service provider after it's torn down and throwing an exception. This doesn't seem to happen 100% of the time though based on the attempts for this workflow run where macOS passes every time, Windows and Linux fail twice, and then all three pass.

Edit: Fails on macOS too in a third repo

Reproduction Steps

Clone martincostello/alexa-london-travel-site@74309e7 and run build.ps1 in the root of the repository.

Expected behavior

The tests pass.

Actual behavior

The tests sometimes crashes with the above error on Linux and Windows.

Regression?

Yes.

Known Workarounds

No response

Configuration

• .NET SDK 8.0.100-rc.1.23410.12
• Microsoft.AspNetCore.* 8.0.0-rc.1.23410.15

Other information

Maybe have been introduced by #89531.

Tagging subscribers to this area: @dotnet/area-extensions-dependencyinjection
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I have some repositories that I've hooked up to ingest the daily builds of .NET 8. In this morning's update I have two repos (one public, one private) where the existing test suite is starting to fail due to this error:

The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.<>c.<.cctor>b__22_0(Object s)
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

It appears that something running in the background for HttpClientFactory is trying to use the service provider after it's torn down and throwing an exception. This doesn't seem to happen 100% of the time though based on the attempts for this workflow run where macOS passes every time, Windows and Linux fail twice, and then all three pass.

Reproduction Steps

Clone martincostello/alexa-london-travel-site@74309e7 and run build.ps1 in the root of the repository.

Expected behavior

The tests pass.

Actual behavior

The tests sometimes crashes with the above error on Linux and Windows.

Regression?

Yes.

Known Workarounds

No response

Configuration

• .NET SDK 8.0.100-rc.1.23410.12
• Microsoft.AspNetCore.* 8.0.0-rc.1.23410.15

Other information

Maybe have been introduced by #89531.

Author:	martincostello
Assignees:	-
Labels:	untriaged, area-Extensions-DependencyInjection
Milestone:	-

[martincostello]

Yeah, looks like it's because of this: https://github.com/dotnet/runtime/pull/89531/files#diff-940dba6ffc5ae548fca7105241869c1a78442c04b968f129ce645407022e83cbR98

/cc @CarnaViire

[CarnaViire]

@martincostello can you please point me to the specific test that triggers the exception? Would it be possible to you to create a standalone minimal repro?

[martincostello]

Sorry I don't know specifically which one it is as it seems to be failing due to a background thread sometime after some test happens to do an HTTP call through HttpClient (and lots of the tests do that) - it's likely one of the "integration" tests that run with an HTTP server which for the first project I've linked to are here.

I can try making a standalone repro, but if you need that I'm not going to be able to have the time to do that until next week.

[martincostello]

Based on the fact all the tests themselves are passing seems like you just need to run "enough" tests for the background timer to fire to do some cleanup after the HTTP server is torn down at the end of the test run, which then kills dotnet test due to the unhandled exception.

[CarnaViire]

I can try making a standalone repro, but if you need that I'm not going to be able to have the time to do that until next week.

No worries. I think I know what happened -- DefaultHttpClientFactory's timer is not disposed when it should be, so it ended up firing when everything including ServiceProvider is already torn down. This was present before, but became prominent since the ILogger object is now Lazy.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I have some repositories that I've hooked up to ingest the daily builds of .NET 8. In this morning's update I have two repos (one public, one private) where the existing test suite is starting to fail due to this error:

The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.<>c.<.cctor>b__22_0(Object s)
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

It appears that something running in the background for HttpClientFactory is trying to use the service provider after it's torn down and throwing an exception. This doesn't seem to happen 100% of the time though based on the attempts for this workflow run where macOS passes every time, Windows and Linux fail twice, and then all three pass.

Edit: Fails on macOS too in a third repo

Reproduction Steps

Clone martincostello/alexa-london-travel-site@74309e7 and run build.ps1 in the root of the repository.

Expected behavior

The tests pass.

Actual behavior

The tests sometimes crashes with the above error on Linux and Windows.

Regression?

Yes.

Known Workarounds

No response

Configuration

• .NET SDK 8.0.100-rc.1.23410.12
• Microsoft.AspNetCore.* 8.0.0-rc.1.23410.15

Other information

Maybe have been introduced by #89531.

Author:	martincostello
Assignees:	CarnaViire
Labels:	area-Extensions-DependencyInjection, area-Extensions-HttpClientFactory
Milestone:	8.0.0