Kubelogin doesn't work correctly with MSI in Container Instances

[AleksandarTomicMs2]

We're want to run run Kubernetes deployments from Container Instances with attached managed identities, but are hitting an issue with MSI authentication. We are running kubelogin in the Managed Service Identity (non interactive) setup, but after the call to kubelogin convert-kubeconfig -l msi, any kubectl command returns the error:

error: You must be logged in to the server (the server has asked for the client to provide credentials)

When running the same script from a VM using the same managed identity, the kubectl commands succeed after running kubelogin. This leads us to believe that our cluster/managed identity setup is correct, but that the issue lies with managed identities in container instances.

[weinong]

chatted offline. your token coming from the IMDS endpoint in Container Instance somehow has spn: prefix, which will definitely fail at the AKS server side which expects <app-id>. We will need to follow up with Container Instance team.

[yvespp]

@weinong was this issue ever resolved? We are running into the exact same problem...

What I noticed is that when the container runs under root (which also happens randomly but is probably an issue of our CI) the aud claim has no spn: prefix and kubectl works.

[weinong]

@yvespp what issue are you seeing? token from IMDS should not have spn:. What you described is an expected scenario.

[yvespp]

I'm doing this inside an ACI container (from our CI/CD):

az login --identity --output none
az account set --subscription $ARM_SUBSCRIPTION_ID
az aks get-credentials --name $AKS_CLUSTER_NAME --resource-group $AKS_CLUSTER_RG --subscription $ARM_SUBSCRIPTION_ID --overwrite-existing

token=$(kubelogin get-token --server-id 6dae42f8-....-....-....-............ --client-id 78d2afd6-....-....-....-............ --login msi -v 99 | jq .status.token -r)
I0204 15:47:13.265529      93 execCredentialPlugin.go:31] Login Method: msi, Environment: AzurePublicCloud, TenantID: , ServerID: 6dae42f8-....-....-....-............, ClientID: 78d2afd6-....-....-....-............, IsLegacy: false, msiResourceID: 
I0204 15:47:13.265633      93 execCredentialPlugin.go:111] acquire new token

The token looks like this:

echo $token | cut -f 2 -d "." | base64 -d
{
  "aud": "spn:6dae42f8-....-....-....-............",
  ...
}

If I try to use the token it returns this:

kubectl get nodes
error: You must be logged in to the server (Unauthorized)

kubectl get nodes --token "$token"
error: You must be logged in to the server (Unauthorized)

But when the ACI container runs as root (which sometimes randomly happens), the spn: prefix is not there and the token works...

The AKS cluster uses the newer AKS-managed AAD integration.

[weinong]

while I'm following up with ACI internal team, can you also open a support request so that we can gather more detail? please send me the SR# via aks-help at service.microsoft.com with reference to this issue.

[yvespp]

@weinong was able to get hold of ACI engineering team. The issue is due to the library in one of their ACI infrastructures. They expect to complete the new infra migration around end of March.

Thanks again @weinong for you help!