missing env var AZURE_TENANT_ID error with Azure App Configuration Kubernetes Provider

[priya30m]

Hi,

I am installing Azure App Configuration Kubernetes Provider by following the steps from:
https://learn.microsoft.com/en-gb/azure/azure-app-configuration/quickstart-azure-kubernetes-service

But installation of pod az-appconfig-k8s-provider is failing with below error.
I0628 09:52:45.197551 1 listener.go:44] "controller-runtime/metrics: Metrics server is starting to listen" addr=":8080"
I0628 09:52:45.197974 1 main.go:80] "setup: starting manager"
I0628 09:52:45.198288 1 internal.go:369] "Starting server" path="/metrics" kind="metrics" addr="[::]:8080"
I0628 09:52:45.198339 1 internal.go:369] "Starting server" kind="health probe" addr="[::]:8081"
I0628 09:52:45.198489 1 leaderelection.go:248] attempting to acquire leader lease azappconfig-system/leader.elect.locker.azappconfig.io...
I0628 09:53:01.766638 1 leaderelection.go:258] successfully acquired lease azappconfig-system/leader.elect.locker.azappconfig.io
I0628 09:53:01.767031 1 controller.go:186] "Starting EventSource" controller="azureappconfigurationprovider" controllerGroup="azconfig.io" controllerKind="AzureAppConfigurationProvider" source="kind source: *v1beta1.AzureAppConfigurationProvider"
I0628 09:53:01.767072 1 controller.go:186] "Starting EventSource" controller="azureappconfigurationprovider" controllerGroup="azconfig.io" controllerKind="AzureAppConfigurationProvider" source="kind source: *v1.ConfigMap"
I0628 09:53:01.767114 1 controller.go:186] "Starting EventSource" controller="azureappconfigurationprovider" controllerGroup="azconfig.io" controllerKind="AzureAppConfigurationProvider" source="kind source: *v1.Secret"
I0628 09:53:01.767163 1 controller.go:194] "Starting Controller" controller="azureappconfigurationprovider" controllerGroup="azconfig.io" controllerKind="AzureAppConfigurationProvider"
I0628 09:53:01.868084 1 controller.go:228] "Starting workers" controller="azureappconfigurationprovider" controllerGroup="azconfig.io" controllerKind="AzureAppConfigurationProvider" worker count=1
I0628 09:53:01.868260 1 appconfigurationprovider_controller.go:96] Start reconcile AzureAppConfigurationProvider "appconfigurationprovider-sample" in "default" namespace
E0628 09:53:02.869511 1 appconfigurationprovider_controller.go:355] "Fail to sync configuration from AppConfiguration to cluster." err=<
DefaultAzureCredential: failed to acquire a token.
Attempted credentials:
EnvironmentCredential: missing environment variable AZURE_TENANT_ID
ManagedIdentityCredential: IMDS token request timed out
AzureCLICredential: fork/exec /bin/sh: no such file or directory

Where should the env var AZURE_TENANT_ID_ be set in order to resolve the issue.

[juniwang]

AZURE_TENANT_ID is not required, actually unable to add it directly to the AppConfigProvider spec. DefaultAzureCredential is chain of supported token credentials including EnvironmentCredential, ManagedIdentityCredential , AzureCLICredential and so on. SDK tries them one by one. Missing AZURE_TENANT_ID means EnvironmentCredential is unavailable but SDK will try other credentials in the chain.

ManagedIdentityCredential is expected if you are following the quick start doc. The real issue here is IMDS token request timed out which indicates timeouts requesting AAD token from Instance Metadata service. IMDS is fundamental service running on each VM. Is it possible to login to the VMs to confirm the availability of IMDS: https://learn.microsoft.com/en-us/azure/virtual-machines/instance-metadata-service?tabs=linux#usage. And there is a FAQ in the page: Why is my request timed out for my call to the service? for IMDS trouble shooting. If IMDS is still unavailable, we may need to involve VMSS team for further trouble shooting.

One alternative is to use Service Principal for auth: https://learn.microsoft.com/en-us/azure/azure-app-configuration/reference-kubernetes-provider#use-service-principal.

Also please make sure managed identity is enabled on your VMSS. You can use either system assigned identity or user assigned identity, please see more at: https://learn.microsoft.com/en-us/azure/azure-app-configuration/reference-kubernetes-provider#authentication.

[priya30m]

managed identity is enabled on both userpool and system pool VMSS. I am not sure how to login to VMSS to debug the issue with IMDS.
I also notice that there are errors in NMI logs

 kubectl logs nmi-6h7jw                                                                                                                                
I0630 05:11:40.090730       1 main.go:90] starting nmi process. Version: v1.8.17. Build date: 2023-06-16-16:01.
I0630 05:11:40.090771       1 main.go:103] features for scale clusters enabled
I0630 05:11:40.405644       1 crd.go:448] CRD lite informers started
I0630 05:11:40.405683       1 main.go:117] running NMI in namespaced mode: false
I0630 05:11:40.405769       1 nmi.go:53] initializing in standard mode
I0630 05:11:40.405800       1 probes.go:41] initialized health probe on port 8085
I0630 05:11:40.405851       1 probes.go:44] started health probe
I0630 05:11:40.405985       1 metrics.go:341] registered views for metric
I0630 05:11:40.406147       1 prometheus_exporter.go:21] starting Prometheus exporter
I0630 05:11:40.406161       1 metrics.go:347] registered and exported metrics on port 9090
I0630 05:11:40.406231       1 server.go:127] listening on 127.0.0.1:2579
W0630 05:11:40.427615       1 iptables.go:123] flushing iptables to add aad-metadata custom chains
E0630 05:25:03.355159       1 server.go:427] failed to get matching identities for pod: kube-system/ama-logs-rs-6c699fdd69-gs556, error: context canceled
I0630 05:25:03.355553       1 server.go:239] status (404) took 60060604308 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.244.0.6

But MIC logs seems to be fine

kubectl logs mic-7f6dcf46db-nplh5                                                                                                                     
I0630 05:11:41.295710       1 main.go:114] starting mic process. Version: v1.8.17. Build date: 2023-06-16-16:01
W0630 05:11:41.295770       1 main.go:119] --kubeconfig not passed will use InClusterConfig
I0630 05:11:41.295777       1 main.go:136] kubeconfig () cloudconfig (/etc/kubernetes/azure.json)
I0630 05:11:41.295927       1 main.go:144] running MIC in namespaced mode: false
I0630 05:11:41.295942       1 main.go:148] client QPS set to: 5. Burst to: 5
I0630 05:11:41.295959       1 mic.go:138] starting to create the pod identity client. Version: v1.8.17. Build date: 2023-06-16-16:01
I0630 05:11:41.317278       1 mic.go:144] Kubernetes server version: v1.25.6
I0630 05:11:41.317496       1 cloudprovider.go:117] MIC using user assigned identity: c46f##### REDACTED #####721c for authentication.
I0630 05:11:41.319680       1 probes.go:41] initialized health probe on port 8080
I0630 05:11:41.319689       1 probes.go:44] started health probe
I0630 05:11:41.319794       1 metrics.go:341] registered views for metric
I0630 05:11:41.319851       1 prometheus_exporter.go:21] starting Prometheus exporter
I0630 05:11:41.319873       1 metrics.go:347] registered and exported metrics on port 8888
I0630 05:11:41.319877       1 mic.go:242] initiating MIC Leader election
I0630 05:11:41.319885       1 leaderelection.go:248] attempting to acquire leader lease default/aad-pod-identity-mic...
I0630 05:11:41.333288       1 leaderelection.go:258] successfully acquired lease default/aad-pod-identity-mic
I0630 05:11:41.702112       1 mic.go:315] upgrading the types to work with case sensitive go-client
I0630 05:11:42.071133       1 mic.go:319] type upgrade completed !!
I0630 05:11:42.202786       1 crd.go:457] CRD informers started
I0630 05:11:42.602976       1 pod.go:72] pod cache synchronized. Took 500.885845ms
I0630 05:11:42.602997       1 pod.go:79] pod watcher started !!
I0630 05:11:42.603014       1 mic.go:400] sync thread started.

[juniwang]

OK, you are using AKS pod identity, then it has nothing to do with IMDS. Pod identity intercepts token requests sent to IMDS. The message failed to get matching identities seems like some sort of pod identity configuration error which results in 404 in getting AAD token. Could you please double confirm the pod identity config and open a ticket to AKS team?

E0630 05:25:03.355159       1 server.go:427] failed to get matching identities for pod: kube-system/ama-logs-rs-6c699fdd69-gs556, error: context canceled
I0630 05:25:03.355553       1 server.go:239] status (404) took 60060604308 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.244.0.6

By the way, AKS pod identity is deprecated, it's still available but not recommended for long term use.