[BUG] ManagedIdentityCredential adds ~10 second penalty to default DefaultAzureCredential locally

[afscrome]

Library name and version

Azure.Identity 1.10.2

Describe the bug

When using DefaultAzureCredential, the default behavior is to include ManagedIdentityCredential in the chain. However this adds a 10 second delay when getting a token for the first time.

Whilst this isn't a massive delay, and only happens on the first token request (subsequent requests appear to cache the chained token source used and so bypass ManagedIdentityCredential), this can affect the inner loop of development - a 10 second delay each time you restart your application to debug a change quickly adds up.

Whilst ExcludeManagedIdentityCredential can be disabled to mitigate this, that cannot be done if the application you're using doesn't expose a way to configure this.

Expected behavior

The cost of having ManagedIdentityCredential in the chain should be minimal. (Ideally it be non existant, although I appreciate in practice some delay will be required)

Actual behavior

Looking at logs, it appears that 4 attempts are made to call the Managed Identity endpoint, with ~1s gaps between each retry. Four retries within seconds of each other seems a bit excessive - I can see why one retry is done to avoid any transient network issues, but realistically what are the odds of a 3rd or 4th retry working?

Insanity is doing the same thing over and over and expecting different results

I do get slightly different behavior's here between my work and personal machine. On my personal machine, I see HTTP Timeouts, and a delay of about 10 secs. On my work machine I see A socket operation was attempted to an unreachable network. (169.254.169.254:80), but the delay is closer to 5 secs.

Personal Machine exception

(To follow)

Work Machine

[Error] Azure-Identity: False MSAL 4.54.1.0 MSAL.NetCore .NET 8.0.0-preview.7.23375.6 Microsoft Windows 10.0.22621 [2023-10-16 16:41:03Z - 19e4c08c-ea24-4161-94c6-1907c637e5e6] Exception type: Azure.Identity.CredentialUnavailableException
---> Inner Exception Details
Exception type: System.AggregateException
---> Inner Exception Details
Exception type: Azure.RequestFailedException
---> Inner Exception Details
Exception type: System.Net.Http.HttpRequestException
---> Inner Exception Details
Exception type: System.Net.Sockets.SocketException

   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|282_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
=== End of inner exception stack trace ===

   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
=== End of inner exception stack trace ===

   at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
   at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)   at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
=== End of inner exception stack trace ===

   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
=== End of inner exception stack trace ===

   at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityClient.AuthenticateCoreAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityClient.AppTokenProviderImpl(AppTokenProviderParameters parameters)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.SendTokenRequestToProviderAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.FetchNewAccessTokenAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)

Reproduction Steps

Run the following code twice - once with ExcludeManagedIdentityCredential set to false (default behaviour), and the next with it set to false. Compare the durations reported by each run.

main.cs:

using Azure.Core;
using Azure.Core.Diagnostics;
using Azure.Identity;
using System.Diagnostics;
using System.Diagnostics.Tracing;

using AzureEventSourceListener listener = AzureEventSourceListener.CreateConsoleLogger(EventLevel.LogAlways);
var scopes = new[] { "https://graph.microsoft.com" };

TokenCredential credential = new DefaultAzureCredential(new DefaultAzureCredentialOptions
{
    // Toggle this on / off
    ExcludeManagedIdentityCredential = false
});

var stopwatch = new Stopwatch();
stopwatch.Start();
var token = await credential.GetTokenAsync(new(scopes), default);
stopwatch.Stop();
Console.WriteLine($"⏱️ DURATION: {stopwatch.Elapsed}");

Environment

.NET SDK:
 Version:   8.0.100-preview.7.23376.3
 Commit:    daebeea8ea

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22621
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.100-preview.7.23376.3\

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.0-preview.7.23375.6
  Architecture: x64
  Commit:       65b696cf5e
  RID:          win-x64

.NET SDKs installed:
  6.0.414 [C:\Program Files\dotnet\sdk]
  7.0.306 [C:\Program Files\dotnet\sdk]
  7.0.401 [C:\Program Files\dotnet\sdk]
  8.0.100-preview.7.23376.3 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.0-preview.7.23375.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0-preview.7.23375.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.22 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.0-preview.7.23376.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download
``

[pallavit]

Thank you for reporting the issue. Tagging and routing to the person best able to assist you.
/cc: @schaabs

[christothes]

Hi @afscrome - This is a known issue when ManagedIdentityCredential is in the chain, due to the balance needed between not failing prematurely in deployed environments and failing fast enough in local dev/test environments. Your best bet is to conditionally exclude that credential from the chain based on some locally defined environment variable.

Whilst ExcludeManagedIdentityCredential can be disabled to mitigate this, that cannot be done if the application you're using doesn't expose a way to configure this.

Is the scenario you are describing here for debugging code you don't own that uses DefaultAzureCredential?

duplicate of #29471

[github-actions]

Hi @afscrome. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions]

Hi @afscrome, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[afscrome]

Is the scenario you are describing here for debugging code you don't own that uses DefaultAzureCredential?

Yes, that is correct.

[christothes]

Resolved in https://www.nuget.org/packages/Azure.Identity/1.11.3