ConnectWithManagedIdentity API throws AzureServiceTokenProviderException when running in .Net Core Azure Service Fabric microservice

[MedAnd]

Actual:

Calling ConnectWithManagedIdentity throws AzureServiceTokenProviderException when hosted in a stateful Service Fabric .Net Core microservice.

Expected:

Calling ConnectWithManagedIdentity within Service Fabric .Net Core microservices (both stateless and stateful) is fully supported. The ConnectWithManagedIdentity API should integrate with the fabric:/System/ManagedIdentityTokenService Service.

Example code:

Exception:

'fabric:/ServiceFabric.DataProtection/DataProtectionService' reported Error for property 'AppConfigurationHealth'.
Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException: Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried the following 4 methods to get an access token, but none of them worked.
Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried to get token using Managed Service Identity. Access token could not be acquired. MSI ResponseCode: BadRequest, Response: {"error":{"correlationId":"93de0137-6a21-490a-827b-ac45c0bad929","code":"InvalidApiVersion","message":"The api-version '2017-09-01' is not supported. Supported version is '2019-07-01-preview'."}}
Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Visual Studio Token provider file not found at "C:\Windows\system32\config\systemprofile\AppData\Local.IdentityService\AzureServiceAuth\tokenprovider.json"
Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. 'az' is not recognized as an internal or external command,
operable program or batch file.

Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: https://login.microsoftonline.com/common. Exception Message: Tried to get token using Active Directory Integrated Authentication. Access token could not be acquired. get_user_name_failed: Failed to get user nameInner Exception : No mapping between account names and security IDs was done

at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.AppConfiguration.ManagedIdentityConnector.ManagedIdentityConnector.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.AppConfiguration.ManagedIdentityConnector.AzconfigClientFactory.d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationOptions.ConnectWithManagedIdentity(String endpoint)
at AspNetCore.DataProtection.ServiceFabric.DataProtectionService.<>c.b__5_0(AzureAppConfigurationOptions options) in C:\Projects\AspNetCore.DataProtection.ServiceFabric\AspNetCore.DataProtection.ServiceFabric\DataProtectionService.cs:line 51
at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationSource.<>c__DisplayClass2_0.<.ctor>b__0()
at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationSource.Build(IConfigurationBuilder builder)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at AspNetCore.DataProtection.ServiceFabric.Dat [Truncated]

Library versions:

Microsoft.Azure.AppConfiguration.AspNetCore" Version="2.0.0-preview-009470001-12"
Microsoft.ServiceFabric" Version="6.5.664"
Microsoft.ServiceFabric.Services" Version="3.4.664"
Microsoft.ServiceFabric.Services.Remoting" Version="3.4.664"

Links / Related:

#143
#100

[zhenlan]

Hi @MedAnd, I was not aware of the support of managed identity from service fabric according to this, but after a quick search, I found service fabric just had a preview of managed identity support. Just to confirm... is this the same document you followed to enable the managed identity in service fabric? Did you get the managed identity working with Azure services other than App Configuration? Looks like service fabric can only support new deployment for managed identity at this point. Is this the case for you?

[jimmyca15]

The exception you linked has an interesting piece: "The api-version '2017-09-01' is not supported. Supported version is '2019-07-01-preview'." It looks like the service fabric managed identity support is using a different api version. All of the msi token services are supposed to be able to be used for msi so will have to figure out where the disconnect is.

[MedAnd]

Should I raise a Service Fabric support ticket, GitHub issue etc or will you guys coordinate?

[jimmyca15]

It sounds like service fabric implementation is missing an api version to me. So it may be faster if you open a support ticket for Service Fabric.

[jimmyca15]

@MedAnd we're also going to coordinate on our side to see if we can figure it out.

[MedAnd]

Doing some research I've come across similar issues connecting to Key Vault... not sure if related?

Lesson #1: pass “RunAs=App;” in the connectionString parameter of AzureServiceTokenProvider. This way it will not try different modes to obtain a token, and the exception is a bit better.

https://luisfsgoncalves.wordpress.com/2018/09/06/asp-net-core-on-azure-service-fabric-accessing-key-vault-with-managed-identities-and-the-errors-on-the-token-endpoint/

[jimmyca15]

I think I may have found out what the issue is. Adding a reference to this NuGet package may resolve the issue: "Microsoft.Azure.Services.AppAuthentication" Version="1.3.1". It will take me a bit to be able to spend up a repro environment to verify this.

[MedAnd]

Will test below as I've got the environment:

Lesson #2: install/update the latest version of Microsoft.Azure.Services.AppAuthentication. The aforementioned package with the configuration extensions doesn’t depend on the latest version of this one. It happens that there’s an hard-coded URL of the endpoint that is used to get the Vault access token… and that endpoint changed since there was an “old way” of getting those tokens that is being deprecated.

[MedAnd]

Just tested with Microsoft.Azure.Services.AppAuthentication Version="1.3.1" and the same exception as above:

fabric:/ServiceFabric.DataProtection/DataProtectionService' reported Error for property 'AppConfigurationHealth'.
Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException: Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried the following 4 methods to get an access token, but none of them worked.
Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried to get token using Managed Service Identity. Access token could not be acquired. Received a non-retryable error. MSI ResponseCode: BadRequest, Response: {"error":{"correlationId":"ddbcf916-0342-4880-af12-58d3ef7873e0","code":"InvalidApiVersion","message":"The api-version '2017-09-01' is not supported. Supported version is '2019-07-01-preview'."}}
Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Visual Studio Token provider file not found at "C:\Windows\system32\config\systemprofile\AppData\Local.IdentityService\AzureServiceAuth\tokenprovider.json"
Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. 'az' is not recognized as an internal or external command,
operable program or batch file.

Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: https://login.microsoftonline.com/common. Exception Message: Tried to get token using Active Directory Integrated Authentication. Access token could not be acquired. Failed to get user name from the operating system.

at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.d__18.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.AppConfiguration.ManagedIdentityConnector.ManagedIdentityConnector.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.AppConfiguration.ManagedIdentityConnector.AzconfigClientFactory.d__1.MoveNext()
--- End of stack trace

[jimmyca15]

@MedAnd okay, that is the same root cause The api-version '2017-09-01' is not supported. Supported version is '2019-07-01-preview'. Will have to continue our follow up with Service Fabric. It is good that we confirmed the latest "Microsoft.Azure.Services.AppAuthentication" does not work though.