Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent Exceptions when calling DefaultAzureCredential #15013

Closed
thecodetinker opened this issue Sep 9, 2020 · 5 comments
Closed

Intermittent Exceptions when calling DefaultAzureCredential #15013

thecodetinker opened this issue Sep 9, 2020 · 5 comments
Assignees
@schaabs
Labels
Azure.Identity Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@thecodetinker
Copy link

Describe the bug
Since moving to the latest Azure Identity libraries we have been plagued with intermittent Exceptions when calling DefaultAzureCredential.GetTokenAsync. The code works fine 99% of the time, but sometimes we get one of the following two Exceptions. The code also then hangs soon afterwards - we haven't any hard evidence to implicate Azure Identity in this but we have suspicions.

Expected behavior
The code should reliably obtain a credential token from Azure. The code should be thread-safe.

Actual behavior (include Exception or Stack Trace)
The call usually works, but occasionally it throws an Exception with one of the following stack traces. Either Collection was modified:

System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
 at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
 at System.Collections.Generic.Dictionary`2.Enumerator.MoveNext()
 at Azure.Identity.CredentialPipeline.ScopeGroupHandler.Fail(String name, DiagnosticScope& scope, Exception exception)
 at Azure.Identity.CredentialDiagnosticScope.RegisterFailed(Exception ex)
 at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
 at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Identity.DefaultAzureCredential.<GetTokenAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<AuthenticateRequestAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<ProcessCoreAsync>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipeline.<SendRequestAsync>d__10.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__27.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__21`1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.Secrets.SecretClient.<GetSecretAsync>d__8.MoveNext()

Or NullReferenceException:

Azure.Identity.AuthenticationFailedException: DefaultAzureCredential authentication failed: Object reference not set to an instance of an object. ---> System.NullReferenceException: Object reference not set to an instance of an object.
 at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
 at Azure.Identity.CredentialPipeline.ScopeGroupHandler.CreateScope(String name)
 at Azure.Identity.CredentialPipeline.StartGetTokenScope(String fullyQualifiedMethod, TokenRequestContext context)
 at Azure.Identity.EnvironmentCredential.<GetTokenImplAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Identity.EnvironmentCredential.<GetTokenAsync>d__10.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Identity.DefaultAzureCredential.<GetTokenFromSourcesAsync>d__14.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext() --- End of inner exception stack trace ---
 at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
 at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Identity.DefaultAzureCredential.<GetTokenAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<AuthenticateRequestAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<ProcessCoreAsync>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipeline.<SendRequestAsync>d__10.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__27.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__21`1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.Secrets.SecretClient.<GetSecretAsync>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at

To Reproduce
This is the code used to call GetSecretAsync, nothing fancy. It is called from a multi-threaded environment although the result is cached elsewhere to avoid quota / throttling issues on the KeyVault itself.

public async Task<string> GetConnectionStringAsync(CancellationToken cancellationToken = default)
{
  var client = new SecretClient(new Uri(mVaultUri), new DefaultAzureCredential());
  {
    var result = await client.GetSecretAsync(mSecretName, cancellationToken: cancellationToken).ConfigureAwait(false);
    return result.Value.Value;
  }
}

Environment:

  • Name and version of the Library package used: [Azure.Identity 1.2.2, Azure.Security.KeyVault.Secrets 4.1.0]
  • Hosting platform or OS and .NET runtime version (Windows Server 2016 and .NET Framework 4.7.2]
@ghost ghost added needs-triage This is a new issue that needs to be triaged to the appropriate team. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Sep 9, 2020
@jsquire jsquire added Azure.Identity Client This issue points to a problem in the data-plane of the library. needs-team-attention This issue needs attention from Azure service team or SDK team labels Sep 9, 2020
@ghost ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Sep 9, 2020
@jsquire
Copy link
Member

jsquire commented Sep 9, 2020

Thank you for your feedback. Tagging and routing to the team member best able to assist.

AlexanderSher added a commit to AlexanderSher/azure-sdk-for-net that referenced this issue Sep 9, 2020
@AlexanderSher
Fix Azure#15013: Intermittent Exceptions when calling DefaultAzureCre…
7977c69 
…dential
AlexanderSher added a commit that referenced this issue Sep 9, 2020
@AlexanderSher
Fix #15013: Intermittent Exceptions when calling DefaultAzureCredenti…
1618e22 
…al (#15025)
@ie-zero
Copy link

ie-zero commented Sep 10, 2020

The response time for providing a fix for this issue is great. Well done to the team who picked it up.

@thecodetinker
Copy link
Author

Yes, thank you for the quick turnaround. This morning we had a different error, but looks to be similar root cause:

Azure.Identity.AuthenticationFailedException: DefaultAzureCredential authentication failed: Source array was not long enough. Check srcIndex and length, and the array's lower bounds. ---> System.ArgumentException: Source array was not long enough. Check srcIndex and length, and the array's lower bounds.
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Collections.Generic.Dictionary`2.Resize(Int32 newSize, Boolean forceNewHashCodes)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at Azure.Identity.CredentialPipeline.ScopeGroupHandler.CreateScope(String name)
at Azure.Identity.CredentialPipeline.StartGetTokenScope(String fullyQualifiedMethod, TokenRequestContext context)
at Azure.Identity.ManagedIdentityCredential.<GetTokenImplAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Identity.ManagedIdentityCredential.<GetTokenAsync>d__7.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Identity.DefaultAzureCredential.<GetTokenFromSourcesAsync>d__14.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext() --- End of inner exception stack trace ---
at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Identity.DefaultAzureCredential.<GetTokenAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<AuthenticateRequestAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<ProcessCoreAsync>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.HttpPipeline.<SendRequestAsync>d__10.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__27.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__21`1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Security.KeyVault.Secrets.SecretClient.<GetSecretAsync>d__8.MoveNext()

@schaabs
Copy link
Contributor

schaabs commented Sep 11, 2020

@thecodetinker Thanks again for filing this issue and the information provided. This issue should be fixed in Azure.Identity 1.2.3 released today. If you find that you are still encountering this issue after upgrading to 1.2.3 please reopen the issue.

@schaabs schaabs closed this as completed Sep 11, 2020
@thecodetinker
Copy link
Author

We've done some runs with the new code and so far, so good. Thanks for turning this around so quickly, much appreciated!

@github-actions github-actions bot locked and limited conversation to collaborators Mar 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@schaabs schaabs

Labels
Azure.Identity Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jsquire @ie-zero @schaabs @thecodetinker