[OpenId] AKS + AAD integration with Kiali's OpenId

[estevaofay]

I have been trying to get Kiali's OpenID integration. The scenario we want is to have Kiali's dashboard exposed to the development teams in a Public URL and they can log in into Kiali with their Azure AD's credentials.

My setup is like this:
Kiali Operator: v.1.24
AKS: version 1.18.8
We have a AKS + AAD cluster integrated like so:

My Kiali Spec has the following configuration:

cr:
  spec:
    auth:
      strategy: openid
      openid:
        client_id: <myClientId>
        issuer_uri: <myIssuerURI>
        authorization_endpoint: <my/authorizeEndpoint>
        scopes: ["openid", "profile", "email"]

My app registration on AKS has the following manifest with obscured ids

{
	"id": "secretid",
	"acceptMappedClaims": null,
	"accessTokenAcceptedVersion": 2,
	"addIns": [],
	"allowPublicClient": null,
	"appId": "appId",
	"appRoles": [],
	"oauth2AllowUrlPathMatching": false,
	"createdDateTime": "2020-09-25T19:20:08Z",
	"groupMembershipClaims": "SecurityGroup",
	"oauth2AllowIdTokenImplicitFlow": true,
	"oauth2AllowImplicitFlow": true,
	"oauth2Permissions": [],
	"preAuthorizedApplications": [],
	"publisherDomain": "publisherDomain",
	"samlMetadataUrl": null,
	"signInUrl": null,
	"signInAudience": "AzureADMyOrg",
	"tags": [
		"notApiConsumer",
		"webApp"
	],
	"tokenEncryptionKeyId": null,
	...other secret configs...
	
}

Here is where it gets interesting:

I get two possible issuerUris from my Azure App registration on Azure.

In the Overview tab you can get your app Endpoints:

Clicking on Endpoint I get the /.wellknown endpoint which gives me a couple of options:

I can use the issuer-uri given me there or I can use the same authorization endpoint in the issuer-uri slot.

Example authorization endpoint:

https://login.microsoftonline.com/your-id-here/oauth2/v2.0/authorize

Example issuer endpoint:

https://login.microsoftonline.com/your-id-here/v2.0

Through extensive testing we noticed that when using the Authorization endpoint on the issuer-uri field we are using OpenId's Implicit Flow

When using the regular issuer uri we are using the Authorization Code Flow.

I was not able to make neither approach work and will go into detail of each one below:

Implicit Code Flow

To use Implicit Code Flow we use the /authorize endpoint twice, like so:

cr:
  spec:
    auth:
      strategy: openid
      openid:
        client_id: <myClientId>
        issuer_uri: https://login.microsoftonline.com/your-id-here/oauth2/v2.0/authorize
        authorization_endpoint: https://login.microsoftonline.com/your-id-here/oauth2/v2.0/authorize
        scopes: ["openid", "profile", "email"]

Describing the flow:

After Clicking on Log in, we can see through the network tab that kiali redirects us to a /kiali/api/auth/openid_redirect endpoint.

This will redirect us to our IdP where you can log in normally.

After that the IdP will redirect to your callback (in this case http://localhost/kiali)

Here Kiali will have:

id_token
state
session_state

In its form data (Confirming this is Implicit Flow)
however with an unsuccessful login screen with the message

Login unsuccessful: Token is not valid or is expired

I went and looked into the JWT that came back in the id_token field and it has the following structure

{
  "aud": "number-here",
  "iss": "https://login.microsoftonline.com/issuer-id-here/v2.0",
  "iat": number-here,
  "nbf": number-here,
  "exp": number-here,
  "aio": "string-here",
  "email": "my-email-here",
  "hasgroups": "true",
  "name": "Fay, Estevao",
  "nonce": "nonce-here",
  "oid": "oid-here",
  "preferred_username": "my-email-here",
  "rh": "hash value here.",
  "other-fields": "omitted"
}

As we can see this JWT does not have the groups field for us to claim it through Kiali OpenId.

It is necessary because our cluster's RBAC rules (including to our admins) are done by using AKS + AD integration through groups.

I noticed here #3042 that you guys have dismissed groups claim in the past but it would be essential to use AKS + AD integration since RBAC assignments are done directly to User or Group Ids which Kiali is apparently not able to resolve.

Auth Code Flow

To use Auth Code Flow, we need to use the issuer-uri given by us by that same .well-known endpoint in Azure App registration

Sample YAML is like so

cr:
  spec:
    auth:
      strategy: openid
      openid:
        client_id: <myClientId>
        issuer_uri: https://login.microsoftonline.com/your-id-here/v2.0/
        authorization_endpoint: https://login.microsoftonline.com/your-id-here/oauth2/v2.0/authorize
        scopes: ["openid", "profile", "email"]

With this we are using Code Flow described below

Describing the Flow

When we click on login we quickly get logged in through our IdP but get this "Token is not valid or is expired" error with a 401.

{
	"error": "Token is not valid or is expired",
	"detail": "Unauthorized"
}

Debugging through our network tab we can see we first got redirected to the IdP, got redirected back to the callback and then kiali made a request to the authorization endpoint with the Query string parameters

• client_id = myId
• response_type = code
• redirect_uri = http://localhost/kiali
• scope = openid email profile
• nonce = asdolfkjlaskdjf
• state = auishfiohaside

Confirming this is the Auth Code Flow.

However, through the standard OpenId Code Flow, at some point the returned code needs to be sent back to the IdP with the client secret which Kiali's docs briefly mention here:

Register Kiali as a client application in your OpenId Server. Use the root path of your Kiali instance as the callback URL. If the OpenId Server provides you a client secret, or if you manually set a client secret, issue the following command to create a Kubernetes secret holding the OpenId client secret:

kubectl create secret generic kiali --from-literal="oidc-secret=$CLIENT_SECRET" -n $NAMESPACE

Which considering that we know that getting oidc flags from cloud provided kubernetes (specially ones created by tools like terraform) is next to impossible as see here #3084. Also we are using a GitOps Approach to installing Kiali, Istio and the likes, so It would be nice to have a strategy of inserting a client key (if necessary) in a way that is not exposed in our git repositories (but I would understand if it were impossible)

This makes is impossible to setup a simple OpenId connect with Azure AKS since we cannot send the secret back to the IdP (which explaing why i'm getting a 401).

However by using https://oidcdebugger.com/ and Postman I can see that if Kiali were to send the correct secret in this flow, we would get the groups in the JWT id_token like so:

{
  "aud": "aud-id-here",
  "iss": "https://login.microsoftonline.com/id-here/v2.0",
  "iat": 1601670894,
  "nbf": 1601670894,
  "exp": 1601674794,
  "aio": "omittingstuff",
  "email": "Estevao.Fay-ext@my-email.com",
  "groups": [
    "groupId1",
    "groupId2",
    "groupId3"
  ],
  "name": "Fay, Estevao",
  "nonce": "boqb7l0suh6",
  "oid": "65b8a2",
  "preferred_username": "my-email-here"

To which I would be able claim groups in kiali if it were a supported feature.

@israel-hdez Asked for me to open a new issue with this! #3084 So here we go! :D

I don't know if I have provided enough information to go forward. I feel like we are close to a solution since we are able to sign in with azure but we are blocked in the next step (which I assume is to check the user's RBAC credentials).

It is our intention to limit as much access to the cluster as possible. Limiting everyone just to visualize Kiali's resources would be ideal. This way we can slowly and securely add more permissions to the teams that need it.

Thanks!

[israel-hdez]

Through extensive testing we noticed that when using the Authorization endpoint on the issuer-uri field we are using OpenId's Implicit Flow

When using the regular issuer uri we are using the Authorization Code Flow.

Using the same value in issuer_uri and authorization_endpoint is a misconfiguration. For your case, I think the right value for issuer_uri is https://login.microsoftonline.com/your-id-here/v2.0. If you go to the logs of the Kiali pod, I'm quite sure you will find a warning line reading Error when fetching OpenID provider's metadata when using the wrong issuer_uri. If you correctly configure the issuer_uri you even can omit configuration of the authorization_endpoint.

The "authorization code" flow does not work with an invalid issuer_uri, but given you manually provided the authorization_endpoint, it's possible to fallback to the "implicit flow". So, Kiali is using the implicit flow as a fallback in that case.

As we can see this JWT does not have the groups field for us to claim it through Kiali OpenId.
It is necessary because our cluster's RBAC rules (including to our admins) are done by using AKS + AD integration through groups.
I noticed here #3042 that you guys have dismissed groups claim in the past but it would be essential to use AKS + AD integration since RBAC assignments are done directly to User or Group Ids which Kiali is apparently not able to resolve.

I'm not sure what you mean that we dismissed group claims :)
We cannot dismiss claims. That's responsibility of the Identity provider to send the claims. However, token claims can be requested through scopes. So, you may just need to configure the right scopes in order to see the group claims, which is what the user of #3042 did and then was able to work with groups.

I see that in the Kiali CR you are using the default scopes: scopes: ["openid", "profile", "email"]. You may need to change/add the needed scope to receive the groups in the token.

Which considering that we know that getting oidc flags from cloud provided kubernetes (specially ones created by tools like terraform) is next to impossible as see here #3084. Also we are using a GitOps Approach to installing Kiali, Istio and the likes, so It would be nice to have a strategy of inserting a client key (if necessary) in a way that is not exposed in our git repositories (but I would understand if it were impossible)

In fact, even if you could fetch the OIDC flags, I don't think you will find the client key there. AFAIK the client key is not passed as a flag to the API server.

I'm just guessing, but in your App registration page in Azure, try going to this section (I'm re-using and editing your image):

There, you may find the "Client secret" that you need to pass to Kiali. If you do, that means you don't really need to find ways to extract OIDC flags and most likely, you may be able to retrieve the needed secret with a simple az command.

This makes is impossible to setup a simple OpenId connect with Azure AKS since we cannot send the secret back to the IdP (which explaing why i'm getting a 401).

Hmm... So, how are you managing secrets for your other apps?
I think it's "normal" that apps use secrets. Istio also requires you to configure secrets under some setups. I would think there should be a solution. Anyway, using secrets is nothing very particular to Kiali, so I'm just sharing thoughts.

[estevaofay]

Okay I'm trying first to configure Auth code flow.

The Kiali CR is like so:

cr:
  spec:
    auth:
      strategy: openid
      openid:
        client_id: "<my-client-app-Id-here>"
        issuer_uri: "https://login.microsoftonline.com/<my-tenant-id-here>/v2.0"
        username_claim: "email"
        scopes: ["openid", "profile", "email"]

So we can avoid the first misconfiguration there.

There, you may find the "Client secret" that you need to pass to Kiali. If you do, that means you don't really need to find ways to extract OIDC flags and most likely, you may be able to retrieve the needed secret with a simple az command.

I had already attempted to create a secret following the docs

By issuing the command

kubectl create secret generic kiali --from-literal="oidc-secret=<my-secret-with-34-characters>" -n istio-system

The secret that I used is the one provided by azure in the section that you pointed out with the red arrow.

I then proceeded to attempt to login.

with logs as such

Hmm... So, how are you managing secrets for your other apps?
I think it's "normal" that apps use secrets. Istio also requires you to configure secrets under some setups. I would think there should be a solution. Anyway, using secrets is nothing very particular to Kiali, so I'm just sharing thoughts.

We only have another example which our ArgoCD application. In this case it is a secret configured in our pipeline that gets injected when creating ArgoCD's app of apps.
I agree it is normal to use secrets, but I don't see a good way to create clusters, app registrations and enable kiali without manual kubectl intervention or having keys in our repo. That's not a direct criticism of Kiali, but of the way the tools are developing in general. I'd be happy to have it working as it is and evolving in the future to a more elegant way (if there is one lol)

I appreciate your help and the discussion, it has been fun making this integration work

[estevaofay]

Attaching the Header information in the network pane

Could the size of the secret be the problem? the docs are fairly specific about the secret size

If you want to enable usage of the OpenId’s authorization code flow, make sure that the Kiali’s signing key is 16, 24 or 32 bytes long
This secret is not needed if you don’t want Kiali to use the authorization code flow (i.e. if your signing key is neither 16, 24 or 32 bytes long).

[israel-hdez]

@xFayre I'm curious. Did you manually create a new App for Kiali in your AAD panel?

Maybe there is another app that could have been automatically created when you enabled AAD integration for your AKS cluster? If there is, probably you should be re-using that one and setup Kiali there, instead of creating a new client app from scratch.

[estevaofay]

@xFayre I'm curious. Did you manually create a new App for Kiali in your AAD panel?

Yes

Maybe there is another app that could have been automatically created when you enabled AAD integration for your AKS cluster? If there is, probably you should be re-using that one and setup Kiali there, instead of creating a new client app from scratch.

Also yes It is a dynamically created app with the cluster. Considering Terraform, GitOps and our infrastructure, we cannot depend on this app to set up the permissions. This would be the case with anyone using terraform.

[israel-hdez]

Also yes It is a dynamically created app with the cluster. Considering Terraform, GitOps and our infrastructure, we cannot depend on this app to set up the permissions. This would be the case with anyone using terraform.

Well, that's unfortunate as my next suggestion is to forget about that Kiali app that you manually created and, instead, use that dynamically created app. So, in that dynamically created app add the Kiali URL as an allowed callback and, then, setup Kiali with the client-id, client secret, issuer uri, etc of that dynamically created app.

The thing is that openid tokens are targeted to specific "apps". Kiali and AKS must share the same logical "app".

[estevaofay]

In the odd case we can make it work with our user-created app. Can you be more specific as to how to create the client secret? Is the kubectl command correct? how do I check if they are the correct 16, 24 or 32 bytes.
A couple of posts ago I posted the kiali logs with this entry:

W1006 19:09:05.663418 1 openid_auth.go:338] Cannot use OpenId authorization code flow because signing key is not 16, 24 nor 32 bytes long

I'm fairly sure this means Kiali couldn't use my client secret created with the kubectl create generic secret kiali
But I'm using the secret from my App registered in AAD so I can't change it to comply with this constraint?

looking at the official CR there is a field way below with:

login_token
   signing_key: ""

Is this related to the openId flow at all? just wanted to put this theory to rest

[israel-hdez]

Can you be more specific as to how to create the client secret? Is the kubectl command correct?

Well, that kubectl command that is in the kiali website, in the docs, should worke fine:

kubectl create secret generic kiali --from-literal="oidc-secret=$CLIENT_SECRET" -n $NAMESPACE

Just replace $CLIENT_SECRET with the plain-text client-secret that you get from the AAD app page.

looking at the official CR there is a field way below with:

login_token
   signing_key: ""

Is this related to the openId flow at all? just wanted to put this theory to rest

Yes, the message Cannot use OpenId authorization code flow because signing key is not 16, 24 nor 32 bytes long is referring to this signing_key of the Kiali CR. You can customize this signing_key to your liking. This is a string, so 16 bytes long could also be understood as 16 characters long. In the docs we are saying "bytes", as some UTF-8 characters are 2 or more bytes long, which is something to keep in mind.

It's a little weird that this message is appearing in the logs, because the kiali-operator should be creating a random 16 bytes long signing key if you don't provide a customized one.

This signing_key is unrelated to the openid client-secret that you get from the app page from AAD. It's more of an internal Kiali requirement to be able to use the Authentication code flow.

[israel-hdez]

@xFayre I'm fixing the openid docs of the website, as this discussion made me realize that they are wrong: kiali/kiali.io#315

Hopefully, this will make easier for people to get the right setup.

[thomasmRavn]

Hope I'm not butting into this thread but I have the same problem, can see docs were updated saying reuse the App Registration for the cluster but for AKS when you enable AD integration Azure creates a Service Principal to handle authentication. I don't think it creates an app registration at all. Does this mean that AKS AD enabled cluster can't work with Kiali? @xFayre if I am hijacking your thread please let me know and I'll create a new issue.

[israel-hdez]

@thomasmRavn Well, I'm now sure really how Azure works, because I don't use it. So, I can only reply based on what I read on the Internet and docs.

I see there is a "legacy" way to configure AzureAD integration in AKS which involves configuring an app in Azure AD. These are the docs I found: https://docs.microsoft.com/en-us/azure/aks/azure-ad-integration-cli. Not sure it it's really a suitable setup for you both @thomasmRavn @xFayre , but if you are willing to give it a try, perhaps it could work?