Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Azure.Identity: AzureCliCredential times out #14691

Closed
snapfisher opened this issue Aug 29, 2020 · 17 comments · Fixed by #16499
Closed

[BUG] Azure.Identity: AzureCliCredential times out #14691

snapfisher opened this issue Aug 29, 2020 · 17 comments · Fixed by #16499
Assignees
Labels
Azure.Identity Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@snapfisher
Copy link

Describe the bug
Please provide the description of issue you're seeing.

I am making a call to GetBlobContainers() from a BlobServiceClient using the AzureCliCredential. The error I receive is {"The ChainedTokenCredential failed due to an unhandled exception: Azure CLI authentication timed out."}. I am running a C# Azure Function from within Visual Studio

Expected behavior
What is the expected behavior?

This should succeed. I am logged into Azure in a PowerShell 7 window on the same machine. It is all local

Actual behavior (include Exception or Stack Trace)
What is the actual behavior?

Exception thrown: 'Azure.Identity.AuthenticationFailedException' in System.Private.CoreLib.dll
Exception thrown: 'Azure.Identity.AuthenticationFailedException' in System.Private.CoreLib.dll
Exception thrown: 'Azure.Identity.AuthenticationFailedException' in System.Private.CoreLib.dll
Exception thrown: 'Azure.Identity.AuthenticationFailedException' in Azure.Storage.Blobs.dll
Exception thrown: 'Azure.Identity.AuthenticationFailedException' in System.Private.CoreLib.dll
Exception thrown: 'Azure.Identity.AuthenticationFailedException' in Azure.Storage.Blobs.dll
Exception thrown: 'Azure.Identity.AuthenticationFailedException' in System.Private.CoreLib.dll
Exception thrown: 'Azure.Identity.AuthenticationFailedException' in System.Private.CoreLib.dll

{"The ChainedTokenCredential failed due to an unhandled exception: Azure CLI authentication timed out."}
Data: {System.Collections.ListDictionaryInternal}
HResult: -2146233088
HelpLink: null
InnerException: Count = 2
Message: "The ChainedTokenCredential failed due to an unhandled exception: Azure CLI authentication timed out."
SerializationRemoteStackTraceString: null
SerializationStackTraceString: " at Azure.Identity.ChainedTokenCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)\r\n at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueFromCredentialAsync(HttpMessage message, Boolean async, CancellationToken cancellationToken)\r\n at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, Boolean async)\r\n at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)\r\n at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)\r\n at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)\r\n at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)\r\n at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 p
ipeline)\r\n at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)\r\n at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)\r\n at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)\r\n at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)\r\n at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)\r\n at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)\r\n at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)\r\n at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)\r\n at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)\r\n at Azure.Core.Pipeline.HttpPipelinePolicy.Process
Next(HttpMessage message, ReadOnlyMemory1 pipeline)\r\n at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)\r\n at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken)\r\n at Azure.Storage.Blobs.BlobRestClient.Service.ListBlobContainersSegmentAsync(ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Uri resourceUri, String version, String prefix, String marker, Nullable1 maxresults, IEnumerable1 include, Nullable1 timeout, String requestId, Boolean async, String operationName, CancellationToken cancellationToken)\r\n at Azure.Storage.Blobs.BlobServiceClient.GetBlobContainersInternal(String continuationToken, BlobContainerTraits traits, BlobContainerStates states, String prefix, Nullable1 pageSizeHint, Boolean async, CancellationToken cancellationToken)\r\n at Azure.Storage.Blobs.Models.GetBlobContainersAsyncCollection.GetNextPageAsync(String continuationToken, Nullable1 pageSizeHint, Bool ean async, CancellationToken cancellationToken)\r\n at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask1 task)\r\n at Azure.Storage.StorageCollectionEnumerator1.StoragePageable.GetEnumerator()+MoveNext()\r\n at Company.Function.HttpTriggerCSharp1.Run(HttpRequest req, ILogger log) in C:\\Users\\pfisher\\source\\repos\\secretlessvs\\secretlessvs\\Function1.cs:line 42" SerializationWatsonBuckets: null Source: "Azure.Identity" StackTrace: " at Azure.Identity.ChainedTokenCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken) in /_/sdk/identity/Azure.Identity/src/ChainedTokenCredential.cs:line 73\r\n at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.<GetHeaderValueFromCredentialAsync>d__11.MoveNext() in /_/sdk/core/Azure.Core/src/Pipeline/BearerTokenAuthenticationPolicy.cs:line 217\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 180\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 151\r\n at System.Runtime.CompilerServices .TaskAwaiter.ValidateEnd(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 122\r\n at System.Threading.Tasks.ValueTask1.get_Result() in //src/System.Private.CoreLib/shared/System/Threading/Tasks/ValueTask.cs:line 763\r\n at System.Runtime.CompilerServices.ValueTaskAwaiter1.GetResult() in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/ValueTaskAwaiter.cs:line 129\r\n at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.<GetHeaderValueAsync>d__8.MoveNext() in /_/sdk/core/Azure.Core/src/Pipeline/BearerTokenAuthenticationPolicy.cs:line 111\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 180\r\n at System.Runtime .CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 151\r\n at System.Threading.Tasks.ValueTask1.get_Result() in //src/System.Private.CoreLib/shared/System/Threading/Tasks/ValueTask.cs:line 763\r\n at System.Runtime.CompilerServices.ValueTaskAwaiter1.GetResult() in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/ValueTaskAwaiter.cs:line 129\r\n at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.<ProcessAsync>d__6.MoveNext() in /_/sdk/core/Azure.Core/src/Pipeline/BearerTokenAuthenticationPolicy.cs:line 61\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 180\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 151\r\n at System.Runtime.CompilerServices.ValueTaskAwaiter.GetResult() in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/ValueTaskAwaiter.cs:line 47\r\n at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task) in /_/sdk/core/Azure.Core/src/Shared/TaskExtensions.cs:line 62\r\n at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) in //sdk/core/Azure.Core/src/Pipeline/BearerTokenAuthenticationPolicy.cs:line 52\r\n at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) in /_/sdk/core/Azure.Core/src/Pipeline/HttpPipelinePolicy.cs:line 50\r\n at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) in //sdk/core/Azure.Core/sr
c/Pipeline/HttpPipelineSynchronousPolicy.cs:line 19\r\n at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) in /_/sdk/core/Azure.Core/src/Pipeline/HttpPipelinePolicy.cs:line 50\r\n at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() in /_/sdk/core/Azure.Core/src/Pipeline/Internal/RetryPolicy.cs:line 61\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63\r\n at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() in /_/sdk/core/Azure.Core/src/Pipeline/Internal/RetryPolicy.cs:line 94\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices /TaskAwaiter.cs:line 180\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 151\r\n at System.Runtime.CompilerServices.ValueTaskAwaiter.GetResult() in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/ValueTaskAwaiter.cs:line 47\r\n at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task) in /_/sdk/core/Azure.Core/src/Shared/TaskExtensions.cs:line 62\r\n at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) in //sdk/core/Azure.Core/src/Pipeline/Internal/RetryPolicy.cs:line 38\r\n at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) in /_/sdk/core/Azure.Core/src/Pipeline/HttpPipelinePolicy.cs:line 50\r\n at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) in //sdk/core/Azure.Core/src/P
ipeline/HttpPipelineSynchronousPolicy.cs:line 19\r\n at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) in /_/sdk/core/Azure.Core/src/Pipeline/HttpPipelinePolicy.cs:line 50\r\n at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) in //sdk/core/Azure.Core/src/Pipeline/HttpPipelineSynchronousPolicy.cs:line 19\r\n at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) in /_/sdk/core/Azure.Core/src/Pipeline/HttpPipelinePolicy.cs:line 50\r\n at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) in //sdk/core/Azure.Core/src/Pipeline/HttpPipelineSynchronousPolicy.cs:line 19\r\n at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken) in //sdk/core/Azure.Core/src/Pipeline/HttpPipeline.cs:line 81\r\n at Azure.Storage.Blobs.BlobRestClient.Service.d__9.MoveNext() in //sdk/storage/Azure.Storage.Blobs/src/Generated/BlobRestClient.cs:line 530\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in //src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in //src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 180\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in //src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 151\r\n at Azure.Storage.Blobs.BlobServiceClient.d__44.MoveNext() in //sdk/storage/Azure.Storage.Blobs/src/BlobServiceClient.cs:line 644\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in //src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63\r\n at Sys
tem.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in /
/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 180\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in //src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 151\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 563\r\n at Azure.Storage.Blobs.Models.GetBlobContainersAsyncCollection.<GetNextPageAsync>d__5.MoveNext() in /_/sdk/storage/Azure.Storage.Blobs/src/Models/GetBlobContainersAsyncCollection.cs:line 42\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in / _/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 180\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 151\r\n at System.Threading.Tasks.ValueTask1.get_Result() in //src/System.Private.CoreLib/shared/System/Threading/Tasks/ValueTask.cs:line 763\r\n at System.Runtime.CompilerServices.ValueTaskAwaiter1.GetResult() in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/ValueTaskAwaiter.cs:line 129\r\n at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask1 task) in //sdk/core/Azure.Core/src/Shared/TaskExtensions.cs:line 50\r\n at Azure.Storage.StorageCollectionEnumerator`1.StoragePageable.d__5.MoveNext() in //sdk/storage/Azure.Storage.Common/src/Shared/StorageCollectionEnumerator.cs:line 103\r\n at Company.Function.HttpTriggerCSharp1.d__0.MoveNext() in C:\Users\pfis
her\source\repos\secretlessvs\secretlessvs\Function1.cs:line 42"
TargetSite: {Azure.Core.AccessToken GetToken(Azure.Core.TokenRequestContext, System.Threading.CancellationToken)}
_HResult: -2146233088
_data: {System.Collections.ListDictionaryInternal}
_dynamicMethods: null
_exceptionMethod: null
_helpURL: null
_innerException: Count = 2
_ipForWatsonBuckets: 0x00007ffcf37818d5
_message: "The ChainedTokenCredential failed due to an unhandled exception: Azure CLI authentication timed out."
_remoteStackTraceString: null
_source: null
_stackTrace: {sbyte[3072]}
_stackTraceString: null
_watsonBuckets: null
_xcode: -532462766
_xptrs: 0x0000000000000000

To Reproduce
Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)

This is the code I am using. config["UserAssignedIdentity"] is null. config["StorageAccountName"] is a url to a storage account where I am the owner and I am a data reader contributor.

        var config = new ConfigurationBuilder()
            .SetBasePath(Directory.GetCurrentDirectory())
            .AddJsonFile("local.settings.json", optional: true, reloadOnChange: true)
            .AddEnvironmentVariables()
            .Build();

        var existingContainers = new List<string>();

        var credential = new ChainedTokenCredential(
            new ManagedIdentityCredential(string.IsNullOrEmpty(config["UserAssignedIdentity"])
                ? null
                : config["UserAssignedIdentity"]),
            new AzureCliCredential());

        try
        {
            var blobServiceClient = new BlobServiceClient(new Uri(config["StorageAccountName"]), credential);
            var containers = blobServiceClient.GetBlobContainers();
            foreach (var container in containers) //Error occurs here
            {
                existingContainers.Add(container.Name);
            }
        }
        catch (Exception e) //exception always caught here
        {
            return new BadRequestObjectResult(e);
        }

Environment:

  • Name and version of the Library package used: Azure.Storage.Blobs 12.5.1, Azure.Identity 1.2.2, Microsoft.NET.Sdk.Functions 3.0.9

  • Hosting platform or OS and .NET runtime version (dotnet --info output for .NET Core projects): [e.g. Azure AppService or Windows 10 .NET Framework 4.8]

PS C:\Users\pfisher> dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 3.1.401
Commit: 5b6f5e5005

Runtime Environment:
OS Name: Windows
OS Version: 10.0.19041
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\3.1.401\

Host (useful for support):
Version: 3.1.7
Commit: fcfdef8d6b

.NET Core SDKs installed:
1.1.13 [C:\Program Files\dotnet\sdk]
1.1.14 [C:\Program Files\dotnet\sdk]
2.1.602 [C:\Program Files\dotnet\sdk]
2.1.604 [C:\Program Files\dotnet\sdk]
2.1.700 [C:\Program Files\dotnet\sdk]
2.1.701 [C:\Program Files\dotnet\sdk]
2.1.801 [C:\Program Files\dotnet\sdk]
2.2.300 [C:\Program Files\dotnet\sdk]
3.1.300 [C:\Program Files\dotnet\sdk]
3.1.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

  • IDE and version : [e.g. Visual Studio 16.3]
    Visual Studio 16.7.1

Other things to note: I do have access to many subscriptions. However, I sepecifically set the subscription appropriately in the CLI window using az account set

@ghost ghost added needs-triage This is a new issue that needs to be triaged to the appropriate team. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Aug 29, 2020
@jsquire jsquire changed the title [BUG] BlobServiceClient : AzureCliCredential times out [BUG] Azure.Identity: AzureCliCredential times out Aug 31, 2020
@jsquire jsquire added Azure.Identity Client This issue points to a problem in the data-plane of the library. needs-team-attention This issue needs attention from Azure service team or SDK team labels Aug 31, 2020
@ghost ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Aug 31, 2020
@jsquire
Copy link
Member

jsquire commented Aug 31, 2020

Thank you for your feedback. Tagging and routing to the team member best able to assist.

@schaabs
Copy link
Contributor

schaabs commented Aug 31, 2020

@snapfisher Thanks for filing this issue, sorry you're running into this problem. To help troubleshoot the issue, after you have logged into the CLI with this specific account can you try to run the following command:

az account get-access-token --output json --resource https://storage.azure.com

If this command succeeds approximately how long does it take to return?

(Please don't share the output of this command as it will contain an access token which can gain access to your account).

@schaabs schaabs added needs-author-feedback More information is needed from author to address the issue. and removed needs-team-attention This issue needs attention from Azure service team or SDK team labels Aug 31, 2020
@snapfisher
Copy link
Author

snapfisher commented Aug 31, 2020

It succeeds. Between 2.5 and 2.6 seconds (using measure-command)

@ghost ghost added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. labels Aug 31, 2020
@sopelt
Copy link
Contributor

sopelt commented Sep 3, 2020

We are seeing the same issue. I grabbed the az .. get-access-token that is being issued from Azure.Identity and that returns in 2-3 Seconds when executed from a terminal.
I'll grab 1.2.2 sources to have a closer look at what is happening here. Or do you already know more @schaabs ?

@sopelt
Copy link
Contributor

sopelt commented Sep 8, 2020

We further investigated (latest sources of Azure.Identity, Azure CLI 2.11.1, added minimal repo) ... and it seems to be related to the actual AAD tenant. The following seems to be the minimal sample in terms of code:

var credential = new AzureCliCredential();
var token = await credential.GetTokenAsync(new TokenRequestContext(new[] { "https://database.windows.net" }));

I also noticed this possibly related issue.

If I az logout and az login --tenant to a different AAD tenant the calls to GetTokenAsync work as expected. We have not yet identified which condition causes azure cli to hang and how (given that the same command as invoked here works when issued in an interactive/cmd terminal).

Using a proxy it looks like the request to login.microsoftonline.com actually succeeds and the cli then proceeds to time out afterwards. On the second attempt there is no request at all (as the token for the same resource is probably cached).

@Rookian
Copy link

Rookian commented Sep 10, 2020

We are also seeing the same issue under high load when we did performance testing.

Stack:

Azure.Identity.AuthenticationFailedException: DefaultAzureCredential failed to retrieve a token from the included credentials.
- EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
- ManagedIdentityCredential authentication unavailable. No Managed Identity endpoint found.
- Azure CLI authentication timed out.
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)

@sopelt
Copy link
Contributor

sopelt commented Sep 11, 2020

@Rookian if it only happens under high load it is probably a new/second issue.
Is it possible that the credential gets invoked on every request and there is no token caching in place (e.g. you directly use the credential for sql access tokens where caching is your responsibility; compared to using the new storage libraries where caching is handled for you)? Do you have metrics on the rate it gets invoked? Are you using AzureCliCredential in production or is this purely for load testing?

@Rookian
Copy link

Rookian commented Sep 11, 2020

@Rookian if it only happens under high load it is probably a new/second issue.
Is it possible that the credential gets invoked on every request and there is no token caching in place (e.g. you directly use the credential for sql access tokens where caching is your responsibility; compared to using the new storage libraries where caching is handled for you)? Do you have metrics on the rate it gets invoked? Are you using AzureCliCredential in production or is this purely for load testing?

I asked a team mate and this behaviour does not only occur under high load and we already cache the token. We use AzureCliCredential so far only from our developer machines. For now we don't have metrics.

@tkaepp
Copy link

tkaepp commented Sep 23, 2020

We face the same issue, when moving MSAL for Azure Keyvault. We had to migrate because our servers in azure probably ran into a SNAT Port Exhaustion described here. https://github.com/dotnet/extensions/issues/3319 We then updated to the new package Azure.Security.KeyVault.Secrets.

Here we run into the same issue described above. We get a timeout in

output = async ? await processRunner.RunAsync().ConfigureAwait(false) : processRunner.Run();

If we run az account get-access-token --output json --resource https://vault.azure.net in cmd, powershell (as normal user and priviledged) we get the access token. It shows our tenant id, subscription id and an expiry date in the future.

We can temporarly workaround this with the InteractiveBrowserCredentials, but then we have to select our account in the browser every single time when starting our solution.

config.AddAzureKeyVault(new Uri($"https://{secretsOptions.AzureKeyVault}.vault.azure.net/"), new InteractiveBrowserCredential());

Also the DefaultAzureCredentials fails to fallback to the InteractiveBrowserCredential because of this timeout exception in the AzureCliCredential.

if (!options.ExcludeAzureCliCredential)
{
    chain[i++] = factory.CreateAzureCliCredential();
}
if (!options.ExcludeInteractiveBrowserCredential)
{
    chain[i++] = factory.CreateInteractiveBrowserCredential(options.InteractiveBrowserTenantId);
}

@gabrielweyer
Copy link

I'm facing the same issue. After reading through the code I think this is due to how the standard output and error are being redirected. I faced a similar issue a while back where the process would hang. This StackOverflow question offers several potential solutions.

I copied the SDK code in a console application and was able to reproduce the hang. I increased the timeout from 10 seconds to 60 seconds and it didn't help. Instead I adapted the ProcessRunner class to be similar to the first answer and I was able to retrieve the token.

@schaabs
Copy link
Contributor

schaabs commented Sep 30, 2020

The issue is due to a deadlock waiting for the process to exit due to the way ProcessRunner reads the process output as @gabrielweyer suggested above. I have a prototype fix but I still need to do some refining and add testing. I'll link the PR once it's available.

@AliGuemues
Copy link

I am running in the same issue. Any updates on this?

@Soruk
Copy link

Soruk commented Oct 19, 2020

I have just migrated from the old SDK to the new one and I have this error 😞

@klaudiaaaj
Copy link

Any updates?

@NeilMacMullen
Copy link

Not sure why it's closed - just ran into this myself. It's pretty reproducible, nothing to do with load though might be a race condition when multiple requests are being made in parallel.

@Skleni
Copy link

Skleni commented Jun 21, 2021

I just encountered the same problem, no load here either.

@shubhambothara
Copy link

I just encountered the same problem too.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Azure.Identity Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
Development

Successfully merging a pull request may close this issue.