Skip to content

Debugging

Krishnakumar R edited this page Jul 24, 2019 · 7 revisions

Pod Labels

One of the common issues seen is the label not properly assigned to the pod. Here is a quick way to check the label is proper:

$ kubectl get po busybox0 --show-labels
NAME       READY     STATUS    RESTARTS   AGE       LABELS
busybox0   1/1       Running   10         10h       aadpodidbinding=select_it,app=busybox0

VM user identity assignment

When aad pod identity is in action, it would assign the identity to underlying vm where the pod is deployed as per the binding specified. We can check if the identity is assigned by running the following -

with VMAs:

az vm identity show -n <vm name> -g <resource group>

with VMSS:

az vmss identity show -g <resource group> -n <scale set name>

In the output of the above command, the userAssignedIdentities indicate whether the user identity got updated on the vm.

Current system state:

The following commands will present the current state of the system:

  1. kubectl get AzureIdentity --all-namespaces -o yaml This should tell us what are the identities which are present currently.

  2. kubectl get AzureIdentityBinding --all-namespaces -o yaml Shows us the binding which are requested. This also has binding selectors presented.

  3. kubectl get AzureAssignedIdentities --all-namespaces -o yaml If any of the pods match with the binding label, then a new AzureAssignedIdentity object is created. The command will help us understand the state of AzureAssignedIdentity.

Note: Currently we don't have a filter in the get commands. Its recommended to remove the Azure related ids and replace them with tags before reporting on GitHub.

Component logs

MIC maintains a state machine which is triggered every time there is a change in the state of the system - pod getting assigned on a node, a new AzureIdentity getting created, a new AzureIdentityBinding getting created etc. and then creates or deletes AzureAssignedIdentity based on the current state. In case of user assigned identity, it then does the required action to assign the user identity to the underlying vm. The logs from this pod will give us clues on whether a binding match was made, any errors in assigned the identity to vm etc.

NMI is the component which intercepts the network traffic and does the token query operations on behalf of the applications. It queries MIC to figure out which identity should be used for a given pod based on it IP address. The logs from NMI will give clues on whether - the application request for token had resulted in a match of the user identity, the errors it received when trying to fetch the token etc.

Note: Currently we don't have a filter in the logs. Its recommended to remove the Azure related ids and replace them with tags before reporting on GitHub.

Increasing the verbosity of the logs

The deployment specification for the MIC and NMI contains the command line arguments for running the respective binaries. We need to add --v=6 onto those lines to increase the verbosity and redeploy the pods. Here is an example of modified

args:

- "--cloudconfig=/etc/kubernetes/azure.json"

- "--logtostderr"

- "--v=6"

You can also perform the same action by doing an kubectl edit on the pod deployment.

Note: Currently we don't have a filter in the logs. Its recommended to remove the Azure related ids and replace them with tags before reporting on GitHub.

You can’t perform that action at this time.