Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] ManagedIdentityCredential throws OperationCanceledException when using SqlClient #1946

Open
Rookian opened this issue Mar 7, 2023 · 5 comments
Open

[BUG] ManagedIdentityCredential throws OperationCanceledException when using SqlClient #1946

Rookian opened this issue Mar 7, 2023 · 5 comments
Labels
ℹ️ Needs more Info Waiting on additional information
Projects
SqlClient Triage Board

Comments

@Rookian
Copy link

Rookian commented Mar 7, 2023

Library name and version

Azure.Identity 1.6.0, Microsoft.Data.SqlClient 4.1.0

Describe the bug

We receive almost everyday SqlExceptions with a inner OperationCanceledException.

{"level":0,"method":"Azure.Core.CancellationHelper.ThrowOperationCanceledException","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":1,"method":"Azure.Core.CancellationHelper.ThrowIfCancellationRequested","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":2,"method":"Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":3,"method":"Azure.Core.Pipeline.ResponseBodyPolicy+<ProcessAsync>d__5.MoveNext","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":4,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":5,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":6,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":7,"method":"Azure.Core.Pipeline.RedirectPolicy+<ProcessAsync>d__5.MoveNext","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":8,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":9,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":10,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":11,"method":"Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":12,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":13,"method":"Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":14,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":15,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":16,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":17,"method":"Azure.Identity.ManagedIdentitySource+<AuthenticateAsync>d__10.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":18,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":19,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":20,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":21,"method":"Azure.Identity.ManagedIdentityClient+<AuthenticateAsync>d__12.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":22,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":23,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":24,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":25,"method":"Azure.Identity.ManagedIdentityCredential+<GetTokenImplAsync>d__14.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":26,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":27,"method":"Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":28,"method":"Azure.Identity.ManagedIdentityCredential+<GetTokenImplAsync>d__14.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":29,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":30,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":31,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":32,"method":"Azure.Identity.ManagedIdentityCredential+<GetTokenAsync>d__12.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":33,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":34,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":35,"method":"Azure.Identity.DefaultAzureCredential+<GetTokenFromSourcesAsync>d__15.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":36,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":37,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":38,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":39,"method":"Azure.Identity.DefaultAzureCredential+<GetTokenImplAsync>d__13.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":40,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":41,"method":"Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":42,"method":"Azure.Identity.DefaultAzureCredential+<GetTokenImplAsync>d__13.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":43,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":44,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":45,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":46,"method":"Azure.Identity.DefaultAzureCredential+<GetTokenAsync>d__12.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":47,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":48,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":49,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":50,"method":"Microsoft.Data.SqlClient.ActiveDirectoryAuthenticationProvider+<AcquireTokenAsync>d__17.MoveNext","line":0,"assembly":"Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"}
{"level":51,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":52,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":53,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":54,"method":"Microsoft.Data.SqlClient.SqlInternalConnectionTds+<>c__DisplayClass146_1+<<GetFedAuthToken>b__1>d.MoveNext","line":0,"assembly":"Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"}
{"level":55,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":56,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":57,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":58,"method":"Microsoft.Data.SqlClient.SqlInternalConnectionTds.GetFedAuthToken","line":0,"assembly":"Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"}

Expected behavior

Clients might need to be able to increase the timeout or Microsoft should fix the endpoint that is responsible for providing an authentication token for ManagedIdentityCredential.

Actual behavior

It seems like SqlClient tries to get a token from ManagedIdentityCredential, but fails due to a timeout.

Anything we can do here?

Reproduction Steps

Environment

Azure AppService (Linux)
.NET 6.0

See also
@Rookian Rookian mentioned this issue Mar 7, 2023
Closed
@lcheunglci lcheunglci added this to Needs triage in SqlClient Triage Board via automation Mar 7, 2023
@lcheunglci
Copy link
Contributor

lcheunglci commented Mar 7, 2023 •
edited

Hi @Rookian, thanks for bringing this issue to our attention. Microsoft.Data.SqlClient 4.1.0 has reached end of support as of Jan 29, 2023, you may want to consider upgrading to the latest LTS version i.e. 5.1.0, where it also upgrades the dependency for Azure Identity to 1.7.0 and soon 1.8.0 on our next release 5.2.0. As for exposing a timeout for ManagedIdentityCredentials from what I can tell from a brief look at the usage in ActiveDirectoryAuthenticationProvider, the ConnectionTimeout from the connection string seems to passed into token context, so perhaps, you can try increasing the ConnectionTimeout and see if it resolves your issue and let us know.

@lcheunglci
Copy link
Contributor

lcheunglci commented Mar 7, 2023 •
edited

Also, how long is OperationCanceledException thrown after the connection is opened, and what is the connection string that you're using? Also if you can provide us with the call stack for the exception, that would be very helpful. Thanks!

@lcheunglci lcheunglci added ℹ️ Needs more Info Waiting on additional information and removed untriaged labels Mar 7, 2023
@lcheunglci lcheunglci moved this from Needs triage to Needs More Info in SqlClient Triage Board Mar 7, 2023
@Rookian
Copy link
Author

Rookian commented Mar 8, 2023

Connection string:
Server=tcp:***.database.windows.net,1433;Initial Catalog=***;Persist Security Info=False;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Authentication=Active Directory Default;

Command Timeout specified via code:

builder.UseSqlServer(connectionString, x =>
{
    x.CommandTimeout(60 * 1000);
    x.EnableRetryOnFailure();
});

@lcheunglci
Copy link
Contributor

lcheunglci commented Mar 8, 2023 •
edited

Hi @Rookian,

I see that your sample code, you're using CommandTimeout and the AAD token context token uses ConnectTimeout from the connection string or ConnectionTimeout from the SqlConnectionStringBuilder, also I believe the value is in seconds, so you don't need to convert it to milliseconds unless you intend to increase it to ~16 minutes. Also if you don't specify the ConnectTimeout, it also defaults to 15 seconds.

@zheweiwangMicrosoft zheweiwangMicrosoft mentioned this issue Feb 18, 2024
Closed
@zheweiwangMicrosoft
Copy link

Hi @Rookian, did you get this issue resolved? I am also suffering such issue with latest MDS 5.2.0-preview5.24024.3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ℹ️ Needs more Info Waiting on additional information
Projects
SqlClient Triage Board
  
Needs More Info
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Rookian @lcheunglci @zheweiwangMicrosoft