TaskCanceled Exception thrown from Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler class

[karun-verghese]

Dear all,

We have a .NET Core Web API service which makes use of the Authentication middleware readily available with .NET 5.0. In our service we have registered 4 different jwt bearer authentication schemes on startup of the service, (why? - that's because we are in the process of migrating from two older identity providers to two newer ones. For example, we are migrating from Azure AD V1 auth endpoints to the Microsoft Identity Platform (V2)).

In a number of our deployments we see TaskCanceled exceptions with the below stack trace, when certain api's are called. The api's are all protected by an [Authorize] attribute which makes use of the aforementioned auth schemes that are registered at startup

Details of the service:
SDK version: .NET 5.0.401
Base docker image: mcr.microsoft.com/dotnet/aspnet:5.0.10

We always seem to have 2 of these exceptions thrown simultaeneously:

System.Threading.Tasks.TaskCanceledException:
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1+<GetConfigurationAsync>d__24.MoveNext (Microsoft.IdentityModel.Protocols, Version=6.7.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler+<HandleAuthenticateAsync>d__6.MoveNext (Microsoft.AspNetCore.Authentication.JwtBearer, Version=5.0.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler+<HandleAuthenticateAsync>d__6.MoveNext (Microsoft.AspNetCore.Authentication.JwtBearer, Version=5.0.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1+<AuthenticateAsync>d__48.MoveNext (Microsoft.AspNetCore.Authentication, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationService+<AuthenticateAsync>d__14.MoveNext (Microsoft.AspNetCore.Authentication.Core, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authorization.Policy.PolicyEvaluator+<AuthenticateAsync>d__2.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Startup+<>c+<<Configure>b__9_0>d.MoveNext

And the second exception stack trace:

  System.Threading.Tasks.TaskCanceledException:
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1+<GetConfigurationAsync>d__24.MoveNext (Microsoft.IdentityModel.Protocols, Version=6.7.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler+<HandleAuthenticateAsync>d__6.MoveNext (Microsoft.AspNetCore.Authentication.JwtBearer, Version=5.0.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

The troublesome thing is that I can't seem to reproduce this error intentionally. We set up suitable load tests to test the affected api's, but we don't seem to be able to reproduce the failures as part of the load tests, but the errors keep happening in the same environments. As far as I can tell we have configured the middleware correctly as for the majority of our calls we are able to authenticate and authorize correctly (we have automated tests testing various scenarios).

But in certain sporadic cases, we get these exceptions.

[Tratcher]

Those look like they're coming from this line:

aspnetcore/src/Security/Authentication/JwtBearer/src/JwtBearerHandler.cs

Line 95 in c6a270f

_configuration = await Options.ConfigurationManager.GetConfigurationAsync(Context.RequestAborted);

The CancellationToken passed in is HttpContext.RequestAborted which fires if the client disconnects mid operation. If you enable debug level server logs you should see other disconnect related logs around this. Do you have clients that are prone to disconnecting?

[karun-verghese]

@Tratcher thanks for the reply, I am still trying to determine this at the time of typing this message. Thanks for the tip on the debug logs, I will come back if I can find anything. Do you know if there's a way I can simulate a client that disconnects in the middle of the api call?

[Tratcher]

You can make the API call with a client like HttpClient or Curl and then cancel it.

[karun-verghese]

@Tratcher thanks, I did something similar, but I had to adjust the response timeout on my client to 20ms to cancel it and see this error. I am now able to reproduce the error. So am I right in understanding that this is expected behaviour right? The cancellation token is being invoked as the client has cancelled the request?

Can you point me to any articles/resources on how to better handle this scenario? Maybe I need to configure a custom delegate for the OnAuthenticationFailed event while registering my authentication scheme?

[Tratcher]

This is expected behavior, and the exception is mainly noise in the app since the client has already disconnected. I'll talk to @blowdart and we'll see what we can do to avoid that.

[blowdart]

Less noise is good

Thanks for contacting us.

We're moving this issue to the .NET 7 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[Tratcher]

Options:
A) Don't pass the RequestAborted CT to GetConfigurationAsync. Config is cached so even if the client disconnects, completing this operation will still have value. We should make sure there's another timeout though.
B) IdentityModel has some changes in flight to fall back to a last-known-good configuration so validation could proceed even if this call failed.

Thanks for contacting us.

We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[chungvodim]

This piece of code is not locked properly:

public override void RequestRefresh()
        {
            DateTimeOffset now = DateTimeOffset.UtcNow;
            if (_isFirstRefreshRequest)
            {
                _syncAfter = now;
                _isFirstRefreshRequest = false;
            }
            else if (now >= DateTimeUtil.Add(_lastRefresh.UtcDateTime, RefreshInterval))
            {
                _syncAfter = now;
            }
        }

So if the condition (now >= DateTimeUtil.Add(_lastRefresh.UtcDateTime, RefreshInterval)) is satisfied => _syncAfter = now
This reset will allow awaiting threads that are trying to invoke the function GetConfigurationAsync to call await _configRetriever.GetConfigurationAsync.

[chungvodim]

• RequestRefresh: https://github.com/AzureAD/azure-activedirectory-identitymodel-extensions-for-dotnet/blob/9e6f90a9a8032107af6f5c851e9d257f4c6c8fdc/src/Microsoft.IdentityModel.Protocols/Configuration/ConfigurationManager.cs#L187-L199
• GetConfigurationAsync : https://github.com/AzureAD/azure-activedirectory-identitymodel-extensions-for-dotnet/blob/9e6f90a9a8032107af6f5c851e9d257f4c6c8fdc/src/Microsoft.IdentityModel.Protocols/Configuration/ConfigurationManager.cs#L125-L166