Limitation "A maximum of 50 pod identities are allowed for a cluster." for real or just in Preview?

[lucasmaj]

The limitation "A maximum of 50 pod identities are allowed for a cluster" from here: https://docs.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity

... is it just in Preview or is it going to stay?

[chewong]

The limitation only applies when you use the managed experience of AAD Pod Identity (creating AzureIdentity & AzureIdentityBinding with azure-cli). @bcho do you know why the maximum number of pod identities is 50?

If you use the open-source solution, there is no limitation on the number of pod identities you can create. For more questions on the managed experience, you can open an issue at https://github.com/Azure/AKS/issues

[bcho]

hi @lucasmaj, yes, we set this limit for preview stage. Do you have a use case that needs more than 50 pod identities? We want to check with some real world use cases and set a proper limit here (due to VMSS side limitation). cc @miwithro

[lucasmaj]

Thank you for consideration on this topic. @bcho Do you have a use case that needs more than 50 pod identities? No, right now I don't. But I do have a SaaS solution made of 30 microservices, each microservice is a k8s Deployment. (mostly to allow for scaling of each microservice individually) Does each microservice/deployment need unique identity? Hopefully not. Maybe yes.

It is just weird that while a cluster has much higher (> (30 * 100)) limits on number of pods, you would be limited by unique identities? As your cluster grows, namespaces are added, more apps and therefore pods gets added, you might run out of pod identities, and the only answer would be "Sorry, get a new cluster. While you could have more pods, we can't give them new identities." Security considerations would directly limit your cluster size?

Am I missing something? Please advise.

[miwithro]

No, it was just set as a base limit for the preview. We will do some testing to introduce higher levels going forward.

[lucasmaj]

Thanks. Earlier I was trying to say that having any limit would feel weird IMHO. Ideally you would not want situation where "Security considerations/limits would put limit on your cluster growth."

[kwaazaar]

Isn't that a limit per nodepool (VMSS)?

Regarding the limit: we use a release pipeline that creates an identity for every single solution that is deployed to the cluster. So every pod (deployment/job) has its own identity. Functionally they could share identities, but since there was no limit before, we simply create separate ones for each pod. Rights are also assigned to these identities through a Azure Devops release pipeline, so maintenance of the identities is no issue. I doubt we are the only ones taking this approach.

[jeanpbond]

@bcho Do you have a use case that needs more than 50 pod identities?

Yes, we have one identity per microservice and we are planning to deploy between 30-50 microservices per environnement. We have clusters in which we deploy up to 4 environments, so we are planning to use 120-200 identities in some clusters. We use one identity per service to secure access to certain resources that are specific to each microservice (Ex: Key Vault).

[bcho]

@kwaazaar @jeanpbond thanks for the info! Cool use cases, we will re/evaluate these usages in our end.

[dhirschfeld]

Just chiming in here that our deployment model assigns an individual identity per app so that the permissions for each app can be assigned, using the principle of least privilege, only what it actually requires.

Having a single identity for every app which is given privileges to all resources that any app may possibly need is less than ideal from a security PoV. It might be possible to share an identity for apps which require access to exactly resources A, B & C and another for apps which need access to A & D but that would be a bit of a logistical nightmare. It's a very straightforward model to create an identity per app and assign only the permissions it requires.

That said we wouldn't hit the 50 identity limit managed service just yet, but we're only beginning our deployment to AKS so we might eventually require a couple of hundred identities per cluster with the deployment model we're using at the moment.

[ericxw]

Adding to the conversation here as well. We have a number of applications planned to get moved into AKS. I'd like to get away from the model where multiple applications and environments share an identity (seems people do it for convenience, but is terrible for security and diagnostics). In such a case, a limit of 50 at the current base setup of 3 environments per application would limit us to 16 applications total, but in the case we have additional environments, we'll start to feel the pain rather soon. Once we open the flood gates up more to automated testing, things are going to get even more hairy.

This won't be realistic for the long run and so we're hoping to see this limit increased. Curious, are there performance considerations, let's say beyond 1000 or something? Any hard limits should be set based on performance/functionality limitations. An optional soft limit can be added which can be overwritten by customer.

[AvigdorLevy]

Shouldn't it be at least 1 identity per namespace (w/o limitation as namespace r not limited)?

[github-actions]

This issue is stale because it has been open 14 days with no activity. Please comment or this will be closed in 7 days.

[lucasmaj]

Bump

[miwithro]

@lucasmaj we already increased the limit to 200. Do you have a use case for more than that?

[lucasmaj]

@miwithro I currently don't have a use case for 200+ identities per cluster. Thanks.

[tchanos]

@lucasmaj Use Case for 200+ identities: In our AKS Cluster, we can onboard new tenants who each get their own Pod Managed Identity in an AKS cluster - each tenant's pod identity would be assigned a role that limits them to their tenant-specific Azure resource, e.g. a Azure SQL DB. So with a limit of 200, this would mean we can only onboard less than 200 tenants before we're forced to create another AKS cluster. If we need to have multiple pod identities per tenant, then this number goes down even faster.

Would another option to allow 200+ pod identities be to not use az aks pod-identity add, instead just deploy AzureIdentity + AzureIdentityBinding ourselves to avoid this limit?

[github-actions]

This issue is stale because it has been open 14 days with no activity. Please comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been stalled for 21 days with no activity. Feel free to re-open if you are experiencing the issue again.