Multiple assigned identities issue

[IvanovOleg]

Have you

• Read Troubleshooting Guide
• Read Known Issues
• Searched on GitHub issues

Describe the bug
Hello. I am trying to make cermanager work with AzureDNS using DNS01 challenge. My AKS cluster is deployed with MSI enabled. AAD pod identity is deployed in the managed mode and forced namespaces. AzureIdentity and AzureIdentityBinding are created, certmanager pod is deployed with a correct label and in the same namespace. When I try to create a certificate, it fails with the next issue:

I0524 08:32:07.905501       1 server.go:302] fetching token for system assigned MSI
E0524 08:32:07.966448       1 server.go:365] failed to get service principal token for pod:security/security-ops-certmanager-867887fdd4-5pbs9 with error code 403, error: failed to get service principal token, error: failed to refresh token, error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request"}

VMss nodes have multiple identities assigned, but I assume that AzureIdentityBinding should make only one identity available for pod, is that correct?

AAD Pod Identity version
helm 4.1.1 (1.8.0)
Kubernetes version
1.19.9

[aramase]

Is the pod security/security-ops-certmanager-867887fdd4-5pbs9 excepted using AzurePodIdentityException? https://azure.github.io/aad-pod-identity/docs/configure/application_exception/

The client token request is forwarded as is when the client pod is excepted using the AzurePodIdentityException. In that scenario, the client needs to pass the clientID as part of token request. If the pod isn't excepted, then NMI component will default the clientID to the first AzureIdentity matched for the pod in the namespace.

[IvanovOleg]

@aramase There is no exception for that pod. Only one identity is bound to that pod.

certmanager:
  enabled: true
  installCRDs: true
  clusterResourceNamespace: security
  serviceAccount: 
    name: "cert-manager"
  cainjector:
    serviceAccount: 
      name: "cert-manager-cainjector"
    podLabels:
      aadpodidbinding: certman-identity
  webhook:
    serviceAccount:
      name: "cert-manager-webhook"
    podLabels:
      aadpodidbinding: certman-identity
  podLabels:
    aadpodidbinding: certman-identity

aad-pod-identity:
  operationMode: managed
  forceNamespaced: true
  azureIdentities:
    certman-identity:
      type: 0
      resourceID: /subscriptions/***/resourceGroups/***/providers/Microsoft.ManagedIdentity/userAssignedIdentities/***
      clientID: ***
      binding:
        name: certman-identity-binding
        selector: certman-identity

I0524 17:12:56.461578       1 server.go:362] exception pod security/security-ops-certmanager-867887fdd4-5pbs9 token handling
I0524 17:12:56.461582       1 server.go:302] fetching token for system assigned MSI
E0524 17:12:56.511463       1 server.go:365] failed to get service principal token for pod:security/security-ops-certmanager-867887fdd4-5pbs9 with error code 403, error: failed to get service principal token, error: failed to refresh token, error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request"}
I0524 17:12:56.511504       1 server.go:196] status (403) took 49996183 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=192.168.88.143

[aramase]

@IvanovOleg There seems to be an exception based on the logs

I0524 17:12:56.461578 1 server.go:362] exception pod security/security-ops-certmanager-867887fdd4-5pbs9 token handling
I0524 17:12:56.461582 1 server.go:302] fetching token for system assigned MSI

Could you share the output for kubectl get azurepodidentityexception -A -o yaml?

If there is an exception that refers to any of the pod labels, then NMI will just forward the token request without any defaulting for clientID.

[IvanovOleg]

@aramase Thanks fo the hint. I have an exception configured for another app that shares one lable with certmanager. Certmanager works after removal.