May I know Azure workload identity works for aks virtual-node-aci-linux

[SandeepCitigori]

Currently, we are using Azure AD Pod Identity for our aks clusters.
We are exploring Azure AD workload identity and able to install using the below links and see it works in nodes as per the given example.
https://azure.github.io/azure-workload-identity/docs/introduction.html
https://github.com/Azure/azure-workload-identity

As mentioned in the 7th point or end of this page, https://azure.github.io/azure-workload-identity/docs/quick-start.html

I have deployed the workload and am able to access the key vault secret via the azure token.

Then I have updated the yaml with nodeSelector & tolerations to install in the virtual-node-aci-linux. Upon deploy, the pod is running but not able to access the keyvault secret.

Deploy pod:
apiVersion: v1
kind: Pod
metadata:
  name: quick-start-vk
  namespace: ${SERVICE_ACCOUNT_NAMESPACE}
spec:
  serviceAccountName: ${SERVICE_ACCOUNT_NAME}
  containers:
    - image: ghcr.io/azure/azure-workload-identity/msal-go:latest
      name: oidc
      env:
      - name: KEYVAULT_NAME
        value: ${KEYVAULT_NAME}
      - name: SECRET_NAME
        value: ${KEYVAULT_SECRET_NAME}
  nodeSelector:
    kubernetes.io/role: agent
    beta.kubernetes.io/os: linux
    type: virtual-kubelet
  tolerations:
    - key: virtual-kubelet.io/provider
      operator: Exists
    - key: azure.com/aci
      effect: NoSchedule

Error when running kubectl logs podname:
E0215 15:21:21.216383 1 token_credential.go:43] 'failed to read the service account token from the filesystem' err='open /var/run/secrets/azure/tokens/azure-identity-token: no such file or directory'

[chewong]

We haven't tried running it in a virtual node but I can give it a try.

Meanwhile, could you confirmed that the pod has been mutated (i.e. kubectl describe pod quick-start-vk -n ${SERVICE_ACCOUNT_NAMESPACE} and check for injected properties described in https://azure.github.io/azure-workload-identity/docs/installation/mutating-admission-webhook.html)

[SandeepCitigori]

I think yes

kubectl describe pod quick-start-1 -n default
Name:         quick-start-1
Namespace:    default
Priority:     0
Node:         virtual-node-aci-linux/
Start Time:   Thu, 17 Feb 2022 07:57:25 -0500
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           zzz.zz.104.4
IPs:
  IP:  zzz.zz.104.4
Containers:
  oidc:
    Container ID:   aci://3ff8ade75zzzzzzzzzzzzzzzzzzzzzzaae59de1f016293
    Image:          ghcr.io/azure/azure-workload-identity/msal-go:latest
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 17 Feb 2022 07:57:25 -0500
    Ready:          True
    Restart Count:  283
    Environment:
      KEYVAULT_NAME:               azwi-kv-tmp-0e1c
      SECRET_NAME:                 tmp-azwi-secret
      AZURE_CLIENT_ID:             zzzzzzz-9zz9-4zze-zzzz-afazzzzzbd6
      AZURE_TENANT_ID:             
      AZURE_FEDERATED_TOKEN_FILE:  /var/run/secrets/azure/tokens/azure-identity-token
      AZURE_AUTHORITY_HOST:        https://login.microsoftonline.com/
    Mounts:
      /var/run/secrets/azure/tokens from azure-identity-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from workload-identity-sa-token-bmlsv (ro)
Conditions:
  Type           Status
  Ready          True 
  Initialized    True 
  PodScheduled   True 
Volumes:
  workload-identity-sa-token-bmlsv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  workload-identity-sa-token-bmlsv
    Optional:    false
  azure-identity-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3600
QoS Class:                   BestEffort
Node-Selectors:              beta.kubernetes.io/os=linux
                             kubernetes.io/role=agent
                             type=virtual-kubelet
Tolerations:                 azure.com/aci:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
                             virtual-kubelet.io/provider op=Exists
Events:                      <none>

[aramase]

This might be a bug in virtual-kubelet/azure-aci with service account token projected volumes. I see projected volume support was added as part of this PR. The check for service account projected volume assumes the token is stored in a Kubernetes secret: https://github.com/virtual-kubelet/azure-aci/blob/master/provider/aci.go#L1740-L1745 which isn't really the case. While default service account token is stored in a Kubernetes secret, the projected service account JWT token are not.

@feiskyer for azure-aci

[SandeepCitigori]

Hi, @chewong did you get a chance to try in virtual node?
I am a bit confused with the above comment, the problem is with azure-aci, so it's already fixed and we can update the azure-aci with the latest or wait?
Please mention if I missed anything.

[aramase]

I am a bit confused with the above comment, the problem is with azure-aci, so it's already fixed and we can update the azure-aci with the latest or wait?

The bug still exists in azure-aci. The PR I linked above added support for projected service account token volumes, but the projected service account token volume is incorrect. I would recommend opening an issue here.

[aramase]

The bug is being tracked in azure-aci: virtual-kubelet/azure-aci#189.

Closing this issue now. Please feel free to reopen if you've any questions.