Azure: Remove a need to provide secret while deploying CAA

[surajssd]

In this new approach of not needing to have secrets, we will still create an identity like we do here but instead of passing the secret AZURE_CAA_CLIENT_SECRET to the daemonset, we will assign an identity to the worker nodes or to the CAA daemonset (workload) itself.

Current Approach

In current approach we create a service principal and use the client_id, client_secret produced out of it and give it to the CAA daemonset.

Advantages

• Simpler works on any type of kubernetes be it a bring-your-own kubernetes or on AKS (of course it has to be on Azure).

Disdvantages

• Have to manage a secret.
• Does not work in CI set up we are trying to build.

Assign identity to the VM

Here we will create an identity and assign it to the worker nodes. So any request originating from the worker node will have permissions to create peer pod VM.

Advantages

• No need to manage a client_secret.
• It will work with CI.
• Regardless of the k8s cluster type this will work. It will work for AKS and any DIY k8s setup on Azure.

Disadvantages

• Obviously this allows "any" workload to create peer pod vm in the same RG.

Assign identity to the workload

Here you can see how one can assign identity to the k8s workloads to be able to communicate with the Azure APIs.

Advantages

• No need to manage a client_secret.
• It will work with CI.
• Only the CAA pods will have that permission to create peer pod VMs.

Disadvantages

• Won't work outside AKS, unless we figure out a way to provision an OIDC issuer on any k8s cluster running on Azure.

[surajssd]

We can support multiple approaches but we have to choose one (or more) to ensure things work on CI and for users.

[bpradipt]

This is a great idea. We can support both the options - viz assigning identity to CAA (and peerpod-ctrl - since this does cleanups for VMs if required) and passing client_secret.

[surajssd]

So there is a way to assign identity to self-managed clusters on Azure: https://azure.github.io/azure-workload-identity/docs/topics/self-managed-clusters.html

I haven't tried it so not sure how this will be translated into our CAA deployments.

[tumberino]

Could you use https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection with your azure clusters? This would be a more cross-provider friendly mechanism?

[surajssd]

Could you use https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection with your azure clusters? This would be a more cross-provider friendly mechanism?

@jtumber-ibm I don't I understand what you mean here. Service account tokens allow you to talk to the k8s api server, so using the k8s RBAC one can define what's allowed.

But here I am talking about how do we allow an application to talk to the Azure API without needing any secret? AKS has a mechanism called workload identity, which is similar to k8s service account token. On the Azure side you define what a particular "workload" on AKS can do with azure APIs and bam, you don't need secret.

My concern was how do we get this working for other DIY k8s clusters on Azure?!

[tumberino]

Could you use https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection with your azure clusters? This would be a more cross-provider friendly mechanism?

@jtumber-ibm I don't I understand what you mean here. Service account tokens allow you to talk to the k8s api server, so using the k8s RBAC one can define what's allowed.

But here I am talking about how do we allow an application to talk to the Azure API without needing any secret? AKS has a mechanism called workload identity, which is similar to k8s service account token. On the Azure side you define what a particular "workload" on AKS can do with azure APIs and bam, you don't need secret.

My concern was how do we get this working for other DIY k8s clusters on Azure?!

Sorry if I wasn't clear. Does Azure not provide a Service Account token to IAM role token endpoint?

[surajssd]

Sorry if I wasn't clear. Does Azure not provide a Service Account token to IAM role token endpoint?

Yes it does but I know this works with the AKS, but not sure how it will work with DIY clusters on Azure. There seems to be a way to do that with DIY clusters, but I haven't tried it myself.