Azure provider does not work when "resource" is used - fails to retrieve email #1144

ludydoo · 2021-03-31T10:45:22Z

on AKS 1.20.2, I'm using oauth2-proxy as a ext-authz extension for istio. I want to protect Kiali. Kiali has multiple authorization strategies, one of them being the header strategy, which would work perfectly with oauth2-proxy. My goal is to forward the kubernetes user token, retrieved using oauth2-proxy from Azure Kubernetes AAD, to kiali, which would then use it to query the kubernetes resources.

I basically want to get an access token to kubernetes through oauth2-proxy, through Azure Kubernetes AD, forward it to Kiali, then kiali would use that to query the Kubernetes API on behalf of the user.

On AKS, there's an integration with Active Directory for RBAC permissions, which is enabled in our clusters. The Kubernetes API is configured with an OpenID provider, being the public Azure Kubernetes AD. So users have to do az aks get-credentials, which guides them through an oidc flow, then configures their kube.config.

I registered an Azure App, which has the Azure Graph user.read permission, as well as the user.read permission on the Azure Kubernetes AD service. The id of this public service is 6dae42f8-4368-4678-94ff-3960e28e3630

I can successfully login trough the istio ext-authz set up on our ingress gateway, then I'm redirected to kiali. But this does not forward the actual kubernetes token to kiali yet.

If I set the oauth2-proxy resource parameter to 6dae42f8-4368-4678-94ff-3960e28e3630 (the Azure Kubernetes AD), then there's an error. The error is that oauth2-proxy cannot retrieve the user email from the graph api (InvalidAudience)

Since the id_token and access_token returned by azure do not include the email, oauth2-proxy tries to retrieve the email from the graph api. This works when the resource parameter is not set, but if fails with an InvalidAudience error if the resource parameter is set.

I tried setting the oauth2-proxy scope arguments to openid email profile or openid,email,profile, but it seems that it does not include the email in the response.

Back to the root of the error, I guess that the reason behind this is that the access token used to query the graph api is the token for the "protected resource" (in this case Azure Kubernetes AD) which does not have permission to query the Graph API.

Expected Behavior

Should not fail to retrieve the email when using a protected resource

Current Behavior

The query fails to retrieve the email from the id_token, access_token and graph api

Possible Solution

Perhaps the Graph API request could be made with a token that was retrieved "not" on the protected resource
Find another way of including the email in the id_token or access_token from Azure

Your Environment

AKS 1.20.2

Version used: latest

 oauth2-proxy [2021/03/31 10:47:07] [azure.go:165] unable to get email claim from id_token: <nil>                                                                                                                                                                  
 oauth2-proxy [2021/03/31 10:47:07] [azure.go:173] unable to get email claim from access token: <nil>                                                                                                                                                              
 oauth2-proxy [2021/03/31 10:47:07] [oauthproxy.go:809] Error creating session during OAuth2 callback: unable to get email address: unexpected status "401": {                                                                                                     
 oauth2-proxy   "error": {                                                                                                                                                                                                                                         
 oauth2-proxy     "code": "InvalidAuthenticationToken",                                                                                                                                                                                                            
 oauth2-proxy     "message": "Access token validation failure. Invalid audience.",                                                                                                                                                                                 
 oauth2-proxy     "innerError": {                                                                                                                                                                                                                                  
 oauth2-proxy       "date": "2021-03-31T10:47:07",                                                                                                                                                                                                                 
 oauth2-proxy       "request-id": "XXXXX",                                                                                                                                                                                          
 oauth2-proxy       "client-request-id": "XXXXXX"                                                                                                                                                                                    
 oauth2-proxy     }                                                                                                                                                                                                                                                
 oauth2-proxy   }                                                                                                                                                                                                                                                  
 oauth2-proxy }

token received with the "resource" param:

{
  "aud": "UUID",
  "iss": "https://sts.windows.net/TENANT_ID/",
  "iat": 2983129083,
  "nbf": 2983129083,
  "exp": 2983129083,
  "aio": "REDACTED",
  "appid": "REDACTED",
  "appidacr": "1",
  "idp": "https://sts.windows.net/TENANT_ID/",
  "oid": "OBJECT_ID",
  "rh": "REDACTED",
  "sub": "REDACTED",
  "tid": "REDACTED",
  "uti": "REDACTED",
  "ver": "1.0"
}

The text was updated successfully, but these errors were encountered:

LouisStAmour · 2021-04-04T16:42:01Z

Not sure if it helps for your use case, but in mine, with the app registration I was using for testing at least, I was able to use Azure AAD OAuth with both v1 and v2 tokens using absolutely no special logic whatsoever from the azure provider - instead I used the generic OIDC provider. Note that my tokens had a lot of default scopes enabled (all of them, for testing purposes). I haven't yet tried using generic Azure app registration tokens with limited scopes. Also note that in the azure app registration Manifest I set the accessTokenAcceptedVersion to "2" for it to actually issue v2 tokens from the v2 endpoint, otherwise it only issues v1 tokens even from the v2 endpoint.

For v1 tokens I used the following (in addition to cookie secrets and other config):

--skip-provider-button
--provider=oidc
--client-id={client-id}
--client-secret={client-secret}
--oidc-issuer-url=https://sts.windows.net/{azure-tenant}/

For v2 tokens I used the following (in addition to cookie secrets and other config):

--skip-provider-button
--provider=oidc
--client-id={client-id}
--client-secret={client-secret}
--oidc-issuer-url=https://login.microsoftonline.com/{azure-tenant}/v2.0

It's not yet clear to me what advantages, if any, there are in using the azure provider directly, especially since it apparently does not validate issuer. #1135 This is fine but not great, as if you're running a multi-tenant install, a better check would be to validate the issuer format matches what is expected, or maybe there's a generic issuer (perhaps v2?) that works in all cases. I don't have a multi-tenant use case currently, and I also don't know if I've set up my token with special scopes or not, though I did basically request all scopes in my token to test out how much data it can return for development purposes.

The advantage of using v1 tokens is that if you use --pass-authorization-header=true (and maybe --pass-basic-auth=false) you can get access to fields that aren't available in v2, such as the amr claim type field which can tell you what form of authentication was used (such as pwd,mfa - rsa is also an option) Upstream you can pull the token from the Authorization header and validate the audience matches your client ID, the issuer matches the issuer noted above, and the JWKS to validate with can be from https://login.microsoftonline.com/{azure-tenant}/discovery/keys for v1 or https://login.microsoftonline.com/{azure-tenant}/discovery/v2.0/keys for v2. (Note that Microsoft doesn't include the "alg" key so PyJWT 2.0.1 has problems, but PyJWT from git upstream, or potentially a newer release, should work fine.)

nodranael · 2021-04-05T16:40:57Z

Hello,

@ludydoo, I'm currently running into similar issues as I'm trying to set up oauth2-proxy to grab an access token to use with the Kubernetes dashboard. @weinong wrote a guide related to this, and implemented the PR (thanks to him, by the way). However, I can't get it to work either.

I'm not familiar with go but by inspecting the code, recompiling and deploying locally, I actually discovered that your current issue is not that the returned id_token does not have an e-mail claim (you can actually configure that in the AAD App registration, in the "Token configuration" section). Based on your logs, you face the same issue I initially had, which is that, in order to extract this claim, the tokens need to be verified, which isn't done unless you configure the issuer with --oidc-issuer-url=https://sts.windows.net/<your tenant id>/. (Trailing slash needed - see #1135 as mentioned by @LouisStAmour).

Once you do that, the tokens will be verified against the AAD OIDC endpoints, and the claims extracted, but now I'm facing the following:

The logic is that the id_token is verified first and checked for an email claim. However, for some reason, the id_token delivered by AAD - in my case, at least - is not signed (the alg header claim is set to none). This causes the validation to fail because of an unsupported algorithm :
unable to verify token: oidc: id token signed with unsupported algorithm, expected ["RS256"] got "none".
@weinong was aware of this, as documented in the code.
Next, we fall back to the access_token which is also verified. However, since we use the --resource flag, the aud claim of this token is not oauth2-proxy's id but rather the requested app id, in my case : the AKS Service (6dae42f8-4368-4678-94ff-3960e28e3630). oauth2-proxy sends the validation with its own client_id, though, so, AAD fails to validate it with
unable to verify token: oidc: expected audience "<my oauth2-proxy app id>" got ["6dae42f8-4368-4678-94ff-3960e28e3630"]
As a last resort, a request is made to the Graph API with the access token, but, once again, it is rejected because the audience is not the one expected by the API :

oauth2-proxy   "error": {                                                                                                                                                                                                                                         
oauth2-proxy     "code": "InvalidAuthenticationToken",                                                                                                                                                                                                            
oauth2-proxy     "message": "Access token validation failure. Invalid audience.",                                                                                                                                                                                 
oauth2-proxy     "innerError": {                                                                                                                                                                                                                                  
oauth2-proxy       "date": "XXXX",                                                                                                                                                                                                                 
oauth2-proxy       "request-id": "XXXXX",                                                                                                                                                                                          
oauth2-proxy       "client-request-id": "XXXXXX"                                                                                                                                                                                    
oauth2-proxy     }                                                                                                                                                                                                                                                
oauth2-proxy   }

So, despite this interesting fallback mechanism, all 3 methods fail - in my case, at least.

Would anyone have some advice in order to get this to work ?

My args:

        - --provider=azure
        - --email-domain=* (will be enforced once the token validation works)
        - --http-address=0.0.0.0:4180
        - --azure-tenant=REDACTED
        - --client-id=<my app id - see weinong's guide>
        - --client-secret=REDACTED
        - --oidc-issuer-url=https://sts.windows.net/<my tenant id>/
        - --scope=user.read
        - --cookie-secret=REDACTED
        - --pass-access-token=true
        - --resource=6dae42f8-4368-4678-94ff-3960e28e3630
        - --set-xauthrequest=true

weinong · 2021-04-06T19:49:52Z

@ludydoo check out my instruction at https://github.com/weinong/k8s-dashboard-with-aks-aad, I verified that it works with my AKS AAD cluster
@LouisStAmour are you using AKD AAD cluster? I believe generic oidc provider will not work with AKS AAD as oidc only establishes who you are, but it doesn't provide the authorization model to access AAD resource (in this case, AKS AAD app)

nodranael · 2021-04-06T20:03:10Z

@weinong , from which token are you able to extract the email claim with your setup ?

weinong · 2021-04-06T20:10:57Z

@nodranael id_token, I understand your scenario, that basically renders the fallback to access token useless in AKS AAD scenario. I'm still trying to think how it can be worked around

nodranael · 2021-04-06T21:36:45Z

well, after removing some token customizations I had made while trying to make this work, I changed --scope=user.read back into --scope=openid and it eventually worked... Not sure why my id_tokens weren't signed before, and now they are signed. Go figure...

mvalenzisi · 2021-04-14T07:02:04Z

I had the same issue and I solved it using --oidc-email-claim=sub.

However, it fails to refresh the cookie with the following error:

[2021/04/13 14:18:33] [stored_session.go:122] Refreshing 1h4m44.989096937s old session cookie for Session{email:p72F5JLVKndLBeNDPZMyn1bUGKuRp_21OAti04fflWY user: PreferredUsername: token:true id_token:true created:2021-04-13 13:13:48.010903063 +0000 UTC expires:2021-04-13 14:13:46 +0000 UTC refresh_token:true} (refresh after 1m0s)
[2021/04/13 14:18:34] [azure.go:303] unable to get email claim from id_token: <nil>
[2021/04/13 14:18:34] [azure.go:256] refreshed id token Session{email:p72F5JLVKndLBeNDPZMyn1bUGKuRp_21OAti04fflWY user: PreferredUsername: token:true created:2021-04-13 14:18:34.128283903 +0000 UTC m=+4106.916933411 expires:2021-04-13 15:18:33 +0000 UTC refresh_token:true} (expired on 2021-04-13 14:13:46 +0000 UTC)

codablock · 2021-04-20T06:45:03Z

I started re-exploring the current Azure AD support through the generic oidc provider yesterday and I can confirm what @LouisStAmour said, as it seems to offer everything that is needed (and more, e.g. groups verification). I only managed to make it work with v2 tokens however.

I gave up on v1 pretty early, so can't say much about chance of success, the only thing I know for sure is that v1 would need the --resource argument to be passed to the authorization endpoint, which is currently only implemented for the Azure provider. The v2 endpoints don't support resource, but instead expect it to be passed as a scope (I only tried it with <app-id>/user_impersonation.

If you miss this part (scope in v2 and resource in v1), access tokens are not validating anymore (as they are only meant for Azure Graph API). If you pass it properly, the access token is not usable anymore to get the profile info (which you have to pass the url first, via --profile-url).

There are currently 2 options to this:

Configure your App Registration to include the email and (optionally) group claims in the ID tokens, this avoids the need for the profile API call
Or oauth2-proxy is modified to support using a different access token for the profile api than passing to the proxied application. Azure AD supports the "On-behalf-of" flow which could be used for this. Not sure if this can be generalised in the oidc provider or if this should be part of the Azure provider.

General question to @weinong @JoelSpeed does it actually make sense to continue supporting the Azure provider? Or does it maybe make sense to drop it and then document how to use the oidc provider for Azure?

JoelSpeed · 2021-05-01T10:58:00Z

In general, where a provider is OIDC compatible, we are looking to deprecate the existing providers and either encourage people to use the OIDC provider directly or convert the existing providers to be extensions of the existing OIDC providers.

So I think documenting using Azure on the OIDC provider would be great

github-actions · 2021-07-01T00:04:08Z

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

pierluigilenoci · 2021-07-01T09:28:46Z

@JoelSpeed I agree with you but to date, Azure is not providing an OIDC compliant provider. Please take a look here especially to my comment here

github-actions · 2021-08-31T00:03:16Z

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

pierluigilenoci · 2021-08-31T08:51:57Z

This is still an issue

JoelSpeed · 2021-09-17T13:04:06Z

@pierluigilenoci What would your proposed route forward be here? We are in the process of refactoring a bunch of providers to rely more heavily on the OIDC provider, reducing our maintenance burden in the long term. We also just added a new keycloakOIDC provider and deprecated the old keycloak provider.

Do you think it's worth looking into building a new OIDC based Azure provider that works with the V2 APIs but can also talk to the graph API to grab the groups?

pierluigilenoci · 2021-09-17T14:07:03Z

@JoelSpeed It is absolutely worth a try.
Whereas hoping that Azure fixes the behavior is quite naive.

github-actions · 2021-11-17T00:03:59Z

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

pierluigilenoci · 2021-11-17T07:57:37Z

This is still an issue

github-actions · 2022-01-17T00:04:33Z

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

pierluigilenoci · 2023-01-24T09:20:20Z

still an issue

paulocuambe · 2023-02-28T11:47:05Z

Do we have any progess on this one?

github-actions · 2023-04-30T00:06:24Z

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

pierluigilenoci · 2023-05-02T11:48:42Z

still an issue

github-actions · 2023-07-03T00:07:41Z

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

pierluigilenoci · 2023-07-03T10:09:06Z

still an issue

github-actions · 2023-09-03T00:06:21Z

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

pierluigilenoci · 2023-09-04T07:08:06Z

still an issue

github-actions · 2023-11-04T00:05:46Z

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

pierluigilenoci · 2023-11-06T17:48:15Z

still an issue

github-actions · 2024-01-06T00:06:21Z

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

github-actions · 2024-03-08T00:06:28Z

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

pierluigilenoci · 2024-03-08T10:57:27Z

@ludydoo, is this still an issue?

luka-mikec · 2024-03-14T13:45:37Z

@pierluigilenoci It is still an issue, or at least was about a month ago which is when I tested

NatiSayada · 2024-04-09T10:34:13Z

I spent too much time trying to resolve this issue, so I hope this summary helps someone.

This is what is working for me now.

oauth2-proxy configuration:

- --provider=oidc
- --provider-display-name=Azure AD
- --email-domain=*
- --http-address=0.0.0.0:4180
- --cookie-secure=true
- --cookie-domain=.your-domain
- --cookie-refresh=59m
- --cookie-expire=1h
- --pass-access-token=true
- --set-xauthrequest=true
- --pass-user-headers=true
- --oidc-email-claim=sub
- --whitelist-domain=.your-domain
- --scope=openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
- --client-id={{.Values.oauth2Proxy.azuread.client_id}}
- --oidc-issuer-url=https://login.microsoftonline.com/{{.Values.oauth2Proxy.azuread.tenant_id}}/v2.0
- --redirect-url=https://{{.Values.hostname}}.{{.Values.domain}}/oauth2/callback
- --skip-claims-from-profile-url=true
- --skip-provider-button=true
- --session-store-type=redis
- --redis-connection-url=redis://redis-service:6379

To make sure that you are getting the right token, go to your instance on /oauth2/auth and check that you are getting the token with the right aud - should be 6dae42f8-4368-4678-94ff-3960e28e3630.
The token should be under the X-Auth-Request-Access-Token header in the auth request.

in my case, I was also having issues with my ingress nginx, as I upgraded the helm version and didn't notice that they changed the default for allowSnippetAnnotations to false as I was using it to pass the token to the Kubernetes dashboard.

This is my ingress annotations:

nginx.ingress.kubernetes.io/auth-response-headers: X-Auth-Request-Access-Token
nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$escaped_request_uri
nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/configuration-snippet: |
  auth_request_set $access_token $upstream_http_x_auth_request_access_token;
  proxy_set_header Authorization "Bearer $access_token";
nginx.ingress.kubernetes.io/proxy-buffer-size: 8k
nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true

This is what versions I'm using:

oauth2-proxy v7.6.0
Kubernetes: 1.29 with Azure RBAC enabled
nginx 1.9.6
nginx helm version: 4.9.1

pierluigilenoci · 2024-04-09T11:03:32Z

@ludydoo, could you please prepone [BUG] for the issue title?

luka-mikec · 2024-04-09T13:58:52Z

@NatiSayada Wow, thanks! It works for me now too. It seems I was missing skip-claims-from-profile-url. I tried it before (I have it commented out in my test configuration), but maybe I tried it in combination with some other parameter that made things fail. Really great to see this finally solved (at least with a workaround) after years with having to stick with 7.2.1

ludydoo changed the title ~~Azure provider does not work when "resource" is used~~ Azure provider does not work when "resource" is used - fails to retrieve email Mar 31, 2021

This was referenced Jun 14, 2021

Investigate and document Kubeapps using Azure managed AAD vmware-tanzu/kubeapps#2943

Closed

Support for Microsoft Identity platform with Azure provider #1231

Closed

github-actions bot added the Stale label Jul 1, 2021

JoelSpeed removed the Stale label Jul 1, 2021

github-actions bot added the Stale label Aug 31, 2021

JoelSpeed removed the Stale label Aug 31, 2021

JoelSpeed added enhancement help wanted labels Sep 17, 2021

github-actions bot added the Stale label Nov 17, 2021

JoelSpeed removed the Stale label Nov 17, 2021

pierluigilenoci mentioned this issue Jan 6, 2022

Azure provider with v7.2.1 and ADAL stop working - Access token validation failure. Invalid audience #1505

Closed

JoelSpeed removed the Stale label Jan 24, 2023

github-actions bot added the Stale label Apr 30, 2023

github-actions bot removed the Stale label May 3, 2023

github-actions bot added the Stale label Jul 3, 2023

github-actions bot removed the Stale label Jul 4, 2023

github-actions bot added the Stale label Sep 3, 2023

github-actions bot removed the Stale label Sep 5, 2023

github-actions bot added the Stale label Nov 4, 2023

github-actions bot removed the Stale label Nov 7, 2023

github-actions bot added the Stale label Jan 6, 2024

tuunit removed the Stale label Jan 7, 2024

github-actions bot added the Stale label Mar 8, 2024

github-actions bot removed the Stale label Mar 10, 2024

Azure provider does not work when "resource" is used - fails to retrieve email #1144

Azure provider does not work when "resource" is used - fails to retrieve email #1144

Comments

ludydoo commented Mar 31, 2021 • edited

Expected Behavior

Current Behavior

Possible Solution

Your Environment

LouisStAmour commented Apr 4, 2021 • edited

nodranael commented Apr 5, 2021 • edited

weinong commented Apr 6, 2021

nodranael commented Apr 6, 2021

weinong commented Apr 6, 2021

nodranael commented Apr 6, 2021

mvalenzisi commented Apr 14, 2021

codablock commented Apr 20, 2021 • edited

JoelSpeed commented May 1, 2021

github-actions bot commented Jul 1, 2021

pierluigilenoci commented Jul 1, 2021

github-actions bot commented Aug 31, 2021

pierluigilenoci commented Aug 31, 2021

JoelSpeed commented Sep 17, 2021

pierluigilenoci commented Sep 17, 2021

github-actions bot commented Nov 17, 2021

pierluigilenoci commented Nov 17, 2021

github-actions bot commented Jan 17, 2022

pierluigilenoci commented Jan 24, 2023

paulocuambe commented Feb 28, 2023

github-actions bot commented Apr 30, 2023

pierluigilenoci commented May 2, 2023

github-actions bot commented Jul 3, 2023

pierluigilenoci commented Jul 3, 2023

github-actions bot commented Sep 3, 2023

pierluigilenoci commented Sep 4, 2023

github-actions bot commented Nov 4, 2023

pierluigilenoci commented Nov 6, 2023

github-actions bot commented Jan 6, 2024

github-actions bot commented Mar 8, 2024

pierluigilenoci commented Mar 8, 2024

luka-mikec commented Mar 14, 2024

NatiSayada commented Apr 9, 2024

pierluigilenoci commented Apr 9, 2024

luka-mikec commented Apr 9, 2024

ludydoo commented Mar 31, 2021 •

edited

LouisStAmour commented Apr 4, 2021 •

edited

nodranael commented Apr 5, 2021 •

edited

codablock commented Apr 20, 2021 •

edited