Enable AAD groups in RBAC

[ams0]

Is this a request for help?:

yes

Is this an ISSUE or FEATURE REQUEST? (choose one):

Feature

What version of acs-engine?:

latest-master

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes

What happened:
I can successfully deploy and RBAC+AAD aware cluster with acs-engine using https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/aad.md . I can assign ClusterRoleBindings to my user, but how can I add an AAD group to a role? It's far more useful is we can map AAD groups to roles in Kubernetes, and then manage who's in those group from AAD.

What you expected to happen:
Allow an entire AAD group to assume a role in Kubernetes

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

[oleborne]

I've looked into this topic and I found the following: the token issued to Kubectl by AAD using Azure authentication plugin does not contain any group information. So while the master is properly configured, it cannot leverage information it does not have access too.

So the limitation is not in ACS-Engine configuration of the masters but in kubectl azure authentication plugin.

I've tried adding the proper group membership claim in my app registration manifest but the token still does not contain the groups.
I don't know if this is a limitation of kubectl azure authentication plugin or a global limitation of the native app authentication flow. I had to interrupt my investigations. I might resume them later.

In theory, if you don't use kubectl azure authentication plugin but obtain a webapp token in a another way, this time include group membership, you should be able to use kubectl base OIDC plugin to pass this token to your api-server. Your api-server would then be able to leverage those groups.

[ams0]

Indeed, from discussions I had on #sig-auth I understood that not always the token returns group information. The problem is exacerbated for users with membership in 200+ groups; in that case the token will return a link to the Graph API that will be needed to be followed to retrieve full group information. I'll keep an eye on the evolution of this, as it's a badly needed feature.

[tuwid]

yeah its a blocker for many environments sadly..

[slack]

Check out https://github.com/Nike-Inc/harbormaster which should be able to add a TokenReview webhook that can call graph to supplement the groups.

[sylr]

Even if we manage to get AAD Groups (either in the token itself or with a subsequent call to the graph API) how would we use them in RBAC ?

Is there a Kind Group we can use in the subjects list of the rbac.authorization.k8s.io/v1beta1 object ?

[mirthy]

I managed to get this to work the AAD Groups. All that needs to be added to the apiserver params is --oidc-groups-claim=groups and then on the AAD setup side in the Server Application Manifest (apiserver), setting groupMembershipClaims from null to either "All" or "SecurityGroup" (See Step 3 in https://azure.microsoft.com/en-us/resources/samples/active-directory-dotnet-webapp-groupclaims/)

Then to add the group rbac: kubectl create clusterrolebinding aad-sample-group-cluster-admin-binding --clusterrole=cluster-admin --group=

Get the group OID from here:
https://portal.azure.com/#blade/Microsoft_AAD_IAM/UserManagementMenuBlade/All groups

I'll work up a PR for the documentation, and code changes. I also found that using --oidc-username-claim=upn or oid work better than sub. I got the idea for upn from here:
https://www.youtube.com/watch?v=giV0WHtp49A

[oleborne]

How do you obtain the token for kubectl? Are you using AAD device login flow using the following option in kubectl: --auth-provider=azure as describe here:
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/plugin/pkg/client/auth/azure/README.md ?

Or did you obtain a token in another way ? (Like a full OAuth flow)

Because I tried exactly what you describe about group membership claim some weeks ago with no success. I’ll try again today. If this works, this is great news.

[mirthy]

I'm using the --auth-provider=azure on a hand built cluster, so I can manipulate group claim. One of the more critical pieces is with the auth-provider=azure piece is the 2 different AAD apps and the one that kubectl uses has to be Native. I got the user/upn claim working first and then I was able to get the group one working. Let me verify again in the morning to make sure I'm not just mistaken.

[oleborne]

SUCCESS!

I found what I did wrong: I was modifying the manifest of the CLIENT app registration and not the SERVER app registration. Which is relevant: the purpose is not to add group information in the token targeted at the CLIENT (ie the id_token) but in the token targeted at the SERVER (ie the access_token).

So the process is to edit the manifest of the app registration for the API SERVER and use either SecurityGroup or All for the groupMembershipClaims that is defaulted to null when creating the app registration. This ensures that the group information (if a user belongs to less than 200 groups) is available in the token. You then need to configure your server to read the groups from the proper claim (the default value is "groups" and this is correct).

Thank you for this solution: I was doing the right thing at the wrong place.