Integration with AKS AAD/RBAC #255
Comments
grolu
added
kind/question
|
Dex as federated OpenID Connect provider cannot work because it also must be configured at the api server in order to work. Since you don't have access to the aks control plane it could never work. You will have to use the ADD/OIDC server connected to your apiserver directly (without dex) in your dashboard configuration. The jwt:
audience: "{{application_id}}"
issuer: "https://sts.windows.net/{{tenant_id}}/"
algorithms: [ RS256 ]
jwks:
strictSsl: true
rejectUnauthorized: true
cache: true
rateLimit: false
jwksRequestsPerMinute: 5
jwksUri: https://login.microsoftonline.com/common/discovery/keys
frontend:
oidc:
authority: "https://sts.windows.net/{{tenant_id}}/"
client_id: "{{application_id}}"
redirect_uri: "{{public_dashboard_url}}/callback"
response_type: 'token id_token'
scope: 'openid email profile groups'
loadUserInfo: falseI hope this hints will help you. Best regards, Holger |
|
Hey, thanks a lot for the pointers, previously we tried to run this with @praveendhac. |
|
A recent trial has been done here: |
|
I am stuck with error, AADSTS50001: Resource identifier is not provided. gardener-dashboard values.yaml Azure side (portal) config Required permissions |
|
It seems that Azure requires a resource to be specified when requesting an access_token. |
|
According to the following documentation https://docs.microsoft.com/de-de/azure/active-directory/develop/v1-protocols-openid-connect-code#send-the-sign-in-request it should not be necessary to specify a resource if the the response_type is only id_token. Currently it is not possible to change the response_type via Helm values.yaml |
|
Using v2.0 worked for below scenario I tried v2.0 but getting Same Origin Policy errors. |
|
We don't pass azure app-registration client secret, we only pass clientid, how is gardener-dashboard going to authenticate with AzureAD. |
|
@praveendhac have you tried user response_type |
|
Tried id_token, got same error "AADSTS50001: Resource identifier is not provided." |
|
@holgerkoser gardener-dashboard AuthN flow is working but don't see anything on the UI. redirect_uri is populating wrong values in configmap.yaml from values.yaml |
|
@praveendhac If you don't see anything in the UI open the developer console. Do you see any errors there e.g. problems with CORS....(if yes which errors do you see). I would also recommend to clear the browser cache. |
|
Yeah, I am getting CSP error (csp:blocked) while accessing https://login.microsoftonline.com/common/discovery/keys |
|
I prepared a PR #268 which should fix the problem that access to the jwksUrl is blocked by CSP rules. It also allows to configure frontend oidc-client via helm. |
|
After the recent problem addressed, we hit several other issues, here are some issues i worked around to see if how far can I go. Apparently even if we solve the "Content Security Policy" and "Access-Control-Allow-Origin" problems seems like my user still seems to be not properly authenticated or either authorizes. Below you can find the steps we took to investigate. 1 - CSP issueAfter clicking the login link in the dashboard we saw this error in the browser console without any logs in the gardener-dashboard pod: And here is a screenshot:
For now i'm able to work around the problem by installing Content-Security-Policy chrome extension and disabling the CSP at browser level. I'm not sure where to address this issue. 2 - CORS misconfigurationAfter working around the CSP issue and tried to login to the dashboard again this we got this one, and again without any logs appearing in the gardener-dashboard pod: And here is a screenshot:
For now, I'm able to work around the problem by installing chrome extension and disabling the CORS at the browser level. I'm not sure where to address this issue. 3 - 401s from dashboard api calls due to wrong authz header sent from browser (likely a bug)After disabling both CSP and CORS on my browser I was able to pass the login sequence but calls to the gardener-dashboard's /api endpoints were getting 401 Checking the requests, i saw that the Authorisation requests are not right, the TODO As a workaround we tampered the connections in the browser and replaced
Example reply from the https://pd-dshboard.ingress.{{cropped}}.io/api/cloudprofiles endpoint. so they work: Here are the logs in gardener-dashboard during the login: 4 - /api/user keeps returning no-admin and cant-create-project dataBut still the https://pd-dshboard.ingress.{{cropped}}.io/api/user endpoint returns no auth
So we tried assigning RBAC rolebinding to the logged-in user: But still no controls in the UI are enabled after logged in and we have no shoots/projects listed. |
ad 1) CSP issueThis should be solved with #268. ad 2) CORS misconfigurationIt seems the azure jwksUri endpoint does not support CORS. ad 3) 401s from dashboard api calls due to wrong authz header sent from browserThis should be solved with #268. ad 4) /api/user keeps returning no-admin and cant-create-project dataI think the username in the token does not match the one confiurged in the bindings. How does the payload of the token look like? You can paste it to https://jwt.io/ for decoding. dashboard image tag :
|
|
@bergerx The problem with jwks endpoint is described here damienbod/angular-auth-oidc-client#19 |
I checked the token, it seems to be matching my k8s username, im not sure if there are some other fields to check, here is how it looks like:
trying with the new image shortly |
|
Just tried the new image, i can confirm that it solves 1 and 3. Thanks a lot for looking into this. For 2:
Can't we do both, first for short/mid term, second for long term For 4: |
|
Also have updated to the new image to check the connect-src issue. Because of this the following also appears. |
|
@eaterm It will not work if you use the default helm value without any change. |
|
@holgerkoser I have changed the values in the helm chart. example.com is just to illustrate the issue. The errors i see have my domain in it. |
|
@bergerx If the value of ad 4) What result do you get if you add a user with the token from the oidc to your kubeconfig and do the following: What do you find in the log of the api-server, why the token is not allowed to create projects? I have create a dev image for the latest commit on the cors branch: |
|
@holgerkoser related to your query, "What ist the oidc configuration of your api-server?". It's AKS cluster, we don't have control of passing OIDC parameters. |
|
@praveendhac I this case I assume it is not |
|
For Roles and Bindings I can use AzureAD Groups/Users as subjects. Should I change this to or create a new role/binding with the Group/Users in AzureAD. I can try different token types which Azure supports. |
|
@holgerkoser Seeing below exception in gardener-dashboard Pod 401 errors |
|
@praveendhac The
|
|
@holgerkoser I am using AKS, the method you are suggesting is directly passing OIDC params to API Server which is not possible in my case as I don't have any control over API Server. |
|
@praveendhac I guess AKS does it differently, see https://github.com/Azure/aks-engine/blob/93caa9ba592d7be5e7d6923ad74adbb2d6348b5c/pkg/api/defaults-apiserver.go#L77 |
|
Using below config as suggested by @holgerkoser not seeing any change in UI behaviour. Debug logs @bergerx I am still unable to figure out This is how I am associating authenticated user/email to cluster only missing part is |
You get a 404 for the keys url? Have you verified that the url is correct? |
|
@praveendhac The PR #270 with the workaround for CORS problem was not merged. Therefor the with the master branch the Of course you have to replace the values for Please do not use |
|
@bergerx @praveendhac Ok. This means they use the field |
|
Added above RBAC with This seems to be another issue,gardener dashboard sends wrong request header |
where do you see this? in chrome's developer tools? |
|
Yes, it's in developer tools logs, the |
|
@praveendhac @bergerx I have merged the PR #268 27 days ago. In this PR I have hardcoded the authorization schema to be |
|
Getting Token looks valid, verified the token on jwt.io replace |
|
The token is valid but your configuration is wrong. The values in the configuration has to exactly match the values in the https://sts.windows.net/{tenantId}/.well-known/openid-configuration. The issuer has trailing slash: If you look carefully in my previous comments you will see that I have proposed the correct URL with trailing slash. You have removed the trailing slash in your config. |
|
@holgerkoser Thanks for the input, all the errors are gone but the link to |
|
Works if email/oid is associated with
|
|
@praveendhac This is not possible. If you use the latest image and your configuration contains What is the resulting frontend configuration of the dashboard. What is the json body of this URL |
|
As I have already written before. The CORS problem has nothing to do with gardener dashboard but it is a bug of the Azure Active Directory OIDC implementation from my point of view. I have provided a workaround which proxies the jwks endpoint from the gardener-dashboard backend to circumvent the Azure CORS problem. As I have written previously the oidc-client we are using allows to specify the oidc metadata directly https://github.com/IdentityModel/oidc-client-js/wiki#provider-settings-if-cors-not-supported-on-oidcoauth2-provider-metadata-endpoint. If the frontend config contain |
|
Below config (helms values.yaml) worked for us. replace To see jwks debug logs in You need to create There are CORS errors and |
|
@praveendhac could you please write a condensed howto on this topic and capture it for everyone under https://gardener.cloud/using-gardener/administrator/ ? This issue when closed will be hard to find for other users running into similar challenges. see the options for including remote content under https://github.com/gardener/documentation . |
|
Previously in gardener/gardener#606 when working on https://gardener.cloud/045_contribute/10_code/30_deploy_seed_into_aks/, I had to left this out since we couldn't deploy dashboard, I'll try to cover this in the AKS installation doc when I have some time. |
|
btw @praveendhac there are some typos in your config. Better recheck all fields thoroughly.
(does it work because any garbled text defaults to false?) |
|
@vasu1124 thanks for catching the typo, config was working even with typo. |
|
@praveendhac I think you don't need this line If certificate validation fails it would be better to add the ca bundle in production. |
gardener-robot-ci-1
added
lifecycle/stale
gardener-robot-ci-1
added
lifecycle/stale
|
As all issues seem to be resolved, I will close this issue. |
TL;DR: OAuth parameter for AKS kube-apiserver's OIDC parameters are not known, and we are not able to configure the Gardener Dashboard the correct way.
We are hosting our Garden and Seed clusters on AKS as described in docs/deployment/aks.md. But with one change. In our Garden AKS cluster now we have AAD enabled. After we wrote that doc AKS went GA and they enabled RBAC and AAD integrations. I created gardener/gardener#551 to update the AKS deployment doc.
Previously we were not able to do that since the we had no control over AKS OIDC parameters. We were not able to deploy the Gardener Dashboard since it requires the Garden Kubernetes cluster authentication work with Oauth token. After AKS went GA and they enabled AAD and RBAC on AKS clusters (https://docs.microsoft.com/en-us/azure/aks/aad-integration), this now enabled us deploy the Dashboard on top AKS. At least we thought that it would.
We enabled AAD on our AKS Garden clusters and started investigating how to configure the Gardener Dashboard to work with AKS. But the Dashboard needs to be configured with the same OIDC parameters with the underlying Kubernetes cluster. But we were on AKS and we don't have visibility on control plane components configuration including the kube-apiserver of the cluster which has the OIDC configuration.
We tried to configure the Gardener with whatever we are able to find in the https://docs.microsoft.com/en-us/azure/aks/aad-integration page create a compatible OIDC parameters for Gardener Dashboard. But we were not able to be successful figure out the right configuration values for the Dashboard.
Next, we tried putting DEX as an intermediary server, Dashboard was using Dex and DEX was authenticating us to the same AAD with the AKS. This time we were able to login to the Dashboard but we were not able to see any resources. This is because the Auth Token provided by DEX is actually not valid in AKS, DEX was kind of proxying the Token.
After spending some time dealing with AAD and DEX configuration we gave up.
The text was updated successfully, but these errors were encountered: