HostedTaskService.StopAsync throws LockRecursionException when using WebApplicationFactory

[MarcelMichau]

Version

What package version of the SDK are you using.

• Nuget package version: 1.0.1
• dll product version: 1.0.1+611dcc0c64

Describe the bug
When running integration tests using WebApplicationFactory for a project which includes an agent added via builder.AddAgent<T>(), during test cleanup/disposal of the WebApplicationFactory, the following exception is thrown:

System.Threading.LockRecursionException
Recursive write lock acquisitions not allowed in this mode.
   at System.Threading.ReaderWriterLockSlim.TryEnterWriteLockCore(TimeoutTracker timeout)
   at System.Threading.ReaderWriterLockSlim.TryEnterWriteLock(TimeSpan timeout)
   at Microsoft.Agents.Hosting.AspNetCore.BackgroundQueue.HostedTaskService.StopAsync(CancellationToken stoppingToken)
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.DisposeAsync()
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.DisposeAsync()
   at MyProject.IntegrationTestFixture.DisposeAsync() in C:\path\to\IntegrationTestFixture.cs:line 67
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90

To Reproduce
I will try & setup a minimal reproduction soon, as I've only experienced this in a larger project. The general steps would be:

1. Use the Empty Agent sample as a base project
2. Add an XUnit Tests Project & configure a simple integration test using the steps outlined here
3. Run the tests & observe that they pass, but test execution continues for a few minutes until the above exception is thrown

Please provide Code Snippets, Channel type, and any special configuration we will need to reproduce this problem.
N/A

Expected behavior
Any Hosted Services setup by builder.AddAgent<T>() should stop gracefully without throwing exceptions when running integration tests with WebApplicationFactory.

Screenshots
N/A

Hosting Information (please complete the following information):

• How are you Hosting this: Local Development Environment - Windows 11 Enterprise
• Are you deploying: N/A
• Are you using Azure Bot Services: N/A
• What Client are you using: N/A
• What .net version is your build in: .NET 9

Additional context
N/A

[MattB-msft]

Thanks for your report @MarcelMichau , can you update to the latest nightly and retest? You can do that by updating your nuget refs to use

1.*-* 

[MarcelMichau]

Thank you @MattB-msft for reverting back.

I've created a minimal repro repo here for reference: https://github.com/MarcelMichau/AgentWebApplicationFactory. This is using the latest nightly build as suggested.

I could not, however, get the same exception thrown in the minimal repro, though I did observe test hangs around 120s during certain test executions.

Here is an example run where the test ran quick:

dotnet test
Restore complete (0,3s)
  EmptyAgent succeeded (0,2s) → X:\Code\Demos\agent-webapplicationfactory\EmptyAgent\bin\Debug\net9.0\EmptyAgent.dll
  EmptyAgentTests succeeded (0,2s) → bin\Debug\net9.0\EmptyAgentTests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.1.0+00ab0a9ec4 (64-bit .NET 9.0.5)
[xUnit.net 00:00:00.09]   Discovering: EmptyAgentTests (app domain = off, method display = ClassAndMethod, method display options = None)
[xUnit.net 00:00:00.11]   Discovered:  EmptyAgentTests (1 test case to be run)
[xUnit.net 00:00:00.13]   Starting:    EmptyAgentTests (parallel test collections = on [24 threads], stop on fail = off, explicit = off)
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://localhost:3978
[xUnit.net 00:00:00.32]   Finished:    EmptyAgentTests
  EmptyAgentTests test succeeded with 4 warning(s) (0,9s)
    C:\Program Files\dotnet\sdk\9.0.300\Microsoft.TestPlatform.targets(48,5): warning : [xUnit.net 00:00:00.13] EmptyAgentTests: Initializing...
    C:\Program Files\dotnet\sdk\9.0.300\Microsoft.TestPlatform.targets(48,5): warning : [xUnit.net 00:00:00.14] EmptyAgentTests: Initialized
    C:\Program Files\dotnet\sdk\9.0.300\Microsoft.TestPlatform.targets(48,5): warning : [xUnit.net 00:00:00.31] EmptyAgentTests: Disposing...
    C:\Program Files\dotnet\sdk\9.0.300\Microsoft.TestPlatform.targets(48,5): warning : [xUnit.net 00:00:00.32] EmptyAgentTests: Disposed

Test summary: total: 1, failed: 0, succeeded: 1, skipped: 0, duration: 0,9s

Here is an example of the test hanging for ~120s:

dotnet test
Restore complete (0,3s)
  EmptyAgent succeeded (0,2s) → X:\Code\Demos\agent-webapplicationfactory\EmptyAgent\bin\Debug\net9.0\EmptyAgent.dll
  EmptyAgentTests succeeded (0,2s) → bin\Debug\net9.0\EmptyAgentTests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.1.0+00ab0a9ec4 (64-bit .NET 9.0.5)
[xUnit.net 00:00:00.07]   Discovering: EmptyAgentTests (app domain = off, method display = ClassAndMethod, method display options = None)
[xUnit.net 00:00:00.09]   Discovered:  EmptyAgentTests (1 test case to be run)
[xUnit.net 00:00:00.10]   Starting:    EmptyAgentTests (parallel test collections = on [24 threads], stop on fail = off, explicit = off)
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://localhost:3978
[xUnit.net 00:02:00.28]   Finished:    EmptyAgentTests
  EmptyAgentTests test succeeded with 4 warning(s) (120,8s)
    C:\Program Files\dotnet\sdk\9.0.300\Microsoft.TestPlatform.targets(48,5): warning : [xUnit.net 00:00:00.10] EmptyAgentTests: Initializing...
    C:\Program Files\dotnet\sdk\9.0.300\Microsoft.TestPlatform.targets(48,5): warning : [xUnit.net 00:00:00.10] EmptyAgentTests: Initialized
    C:\Program Files\dotnet\sdk\9.0.300\Microsoft.TestPlatform.targets(48,5): warning : [xUnit.net 00:00:00.28] EmptyAgentTests: Disposing...
    C:\Program Files\dotnet\sdk\9.0.300\Microsoft.TestPlatform.targets(48,5): warning : [xUnit.net 00:02:00.28] EmptyAgentTests: Disposed

Test summary: total: 1, failed: 0, succeeded: 1, skipped: 0, duration: 120,8s
Build succeeded with 4 warning(s) in 121,9s

Note that the hang does not occur on every run, only on certain runs, which makes it tricky to reproduce reliably.

[tracyboehrer]

@MarcelMichau One thought is that we override the Hosted Service shutdown. But... the default timeout is 60s and that doesn't quite match up.

[MarcelMichau]

Thanks @tracyboehrer - I believe the 120s number is due to the two registered hosted services (HostedActivityService & HostedTaskService) each having a 60s timeout. When the .NET host is stopped, it stops each hosted service sequentially instead of in parallel & that adds up to 120s.

Doing some digging, I'm wondering if the exception is thrown as a result of this issue where the StopAsync method of the hosted service is called a first time where TryEnterWriteLock is called, and then a second time StopAsync is invoked again, at which point the write lock is already entered, hence the LockRecursionException gets thrown.