Teleport AKS Auto Discovery not working for Azure China

[waleed-cariad]

Expected behavior:
Teleport AKS Auto Discovery service, when configured with right acces and roles, should get a token from Azure and discover all AKS clusters it has given access to.

Current behavior:
We are trying to set-up Azure AKS auto-discovery as mentioned here: https://goteleport.com/docs/kubernetes-access/discovery/azure/

After initial set-up, when teleport discovery was enabled, we are seeing the following:

 EnvironmentCredential: missing environment variable AZURE_TENANT_ID

 WorkloadIdentityCredential: no client ID specified. Check pod configuration or set ClientID in the options

 ManagedIdentityCredential: failed to authenticate a system assigned identity. The endpoint responded with {&#34;error&#34;:&#34;invalid_resource&#34;,&#34;error_description&#34;:&#34;AADSTS500011: The resource principal named https://management.core.windows.net/ was not found in the tenant named <tenent name>. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant. Trace ID: ad0eafb3-3849-49fc-b585-1ebdd8cf7000 Correlation ID: ff201dd8-9af0-45fc-9a81-80721772b1ea Timestamp: 2024-01-19 14:25:33Z&#34;,&#34;error_codes&#34;:[500011],&#34;timestamp&#34;:&#34;2024-01-19 14:25:33Z&#34;,&#34;trace_id&#34;:&#34;ad0eafb3-3849-49fc-b585-1ebdd8cf7000&#34;,&#34;correlation_id&#34;:&#34;ff201dd8-9af0-45fc-9a81-80721772b1ea&#34;,&#34;error_uri&#34;:&#34;https://chinanorth2.login.partner.microsoftonline.cn/error?code=500011&#34;}

AzureCLICredential: Azure CLI not found on path
The Azure VM where teleport discovery service is installed is configured with sysrem-assigned identity with right access roles.

To add: the VM where teleport discovery service is runnuing is configured with system-assigned-identity and also role-assignments has been configured for this identity to access AKS clusters.

Our understanding is: for China, the resource principal mentioned in the error above should be "https://management.core.chinacloudapi.cn/" besides "https://management.core.windows.net/"

Bug details:

• Teleport version: Teleport v14.3.0 git:v14.3.0-0-g390d33c go1.21.5
• Debug logs: Provided above

[waleed-cariad]

@AntonAM Hi Anton, can you please provide any update on this? We are currectly blocked by this to roll out our solution on Azure China environments.

[rosstimothy]

The root of the problem seems to be caused by the fact that Teleport isn't setting the Cloud when creating azure clients. This means it will always default to AzurePublic. I imagine the same error would occur if any one tried to deploy in either AzureGovernment or AzureChina.

See Azure/azure-sdk-for-go#21807 for more details.

[AntonAM]

@waleed-cariad can you try running discovery service with environment variable AZURE_AUTHORITY_HOST : https://login.chinacloudapi.cn/ . Also, can you provide teleport logs with a bit of context (a few lines before the error and a few line after error, also with the line numbers and file information if present in the logs).

[waleed-cariad]

@AntonAM I have attached a log file named teleport_discovery_logs for your reference. This is everything we see after running discovery service, please refer to line 85 starting with ManagedIdentityCredential: failed to authenticate a system assigned identity.
About running discovery service with environment variable AZURE_AUTHORITY_HOST, I think this will work only when we use EnvironmentCredential for authentication but we are not using them at all and we don't want to as we want to rely on Azure API to handle all authentications using Managed Identities
teleport_discovery_logs.json
P.S: it still doesn't work though if we set environment variables for AZURE_AUTHORITY_HOST and AZURE_TENANT_ID

[waleed-cariad]

Just to add, as mentioned here on Azure Docs, we are successfully able to get an access token by manually running the following on that Azure VM where Teleport Process is running with the VM being assigned a Managed Identity curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.core.chinacloudapi.cn/'.
By replacing the resource in above request from https://management.core.chinacloudapi.cn/ to https://management.core.windows.net/, we got the following obvious error:

{"error":"invalid_resource","error_description":"AADSTS500011: The resource principal named https://management.core.windows.net/ was not found in the tenant named VGC. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant. Trace ID: a97993dd-1433-4a06-b322-204bb9f42e00 Correlation ID: 2646efeb-ae3b-43f6-83e9-a4a372af9b87 Timestamp: 2024-02-20 09:17:14Z","error_codes":[500011],"timestamp":"2024-02-20 09:17:14Z","trace_id":"a97993dd-1433-4a06-b322-204bb9f42e00","correlation_id":"2646efeb-ae3b-43f6-83e9-a4a372af9b87","error_uri":"https://chinanorth2.login.partner.microsoftonline.cn/error?code=500011"}

This is the same as we are seeing in Teleport Logs when trying to run Teleport Processes to use Azure Managed identity of that VM where Teleport is deployed. Hope that helps.

I guess you still don't support Azure China as mentioned here on line 88 ?

[waleed-cariad]

@AntonAM @rosstimothy Can you please let us know any timeline for this to be fixed? As I mentioned, we are blocked by this to roll-out our solution to China and we can't just do it in Europe only as that does not help us to keep our Architecture replicated in all regions. Also, we are in the process of buying Teleport licence hopefully but we need to make sure first that we can use teleport both in Europe and in China. Can you please help to fix this issue else we will be completely stuck by this.

[zmb3]

Hi @waleed-cariad, this issue is not currently under active development. If you are a Teleport Enterprise customer (or in the process of becoming one) I would encourage you to raise this with your account rep. They'll be able to start some conversations internally that will help us determine when we can schedule this work.