Add Authorization Bearer <jwt> style headers

[JoelSpeed]

This has already been discussed in #530 as a way to allow the proxy to be used in front of applications expecting the raw id token JWT in an authorization header.

Whilst making the changes, I was initially testing with Azure AD as an OIDC provider. Turns out their JWTs are huge! This made the session cookie overflow a single cookie. Which seems like a bit of a flaw that the proxy doesn't handle. So I've also made an attempt at fixing cookies that overflow a single cookie 4kb limit.

To do this I've added some logic to split, index and rejoin cookies if they are too big to fit in a single cookie. In most cases the code will just return the original cookie in a slice, maintaining the original behaviour, but it can handle splitting them if they get too large as demonstrated by the tests added.

I have tested both the cookie splitting and the authorization header changes using the google provider and the OIDC provider.

[hafe]

Have you considered a token refresh in oidc.go::RefreshSessionIfNeeded() ? Like the google provider

[JoelSpeed]

@hafe I'm not sure I see the benefit of this.

Refresh tokens are typically used for offline applications to refresh tokens without having to reopen the browser and get the user to reauthenticate. I didn't think the OAuth proxy was used for that kind of application?

If you're in browser you shouldn't need to refresh the tokens in the OAuth proxy, you are better off letting the browser redirect through the Auth Flow again I would have thought?

[hafe]

for example when I go to strava.com after days or weeks I still don't have to enter any credentials. I can just hit the "login with google" button and I am at the dashboard. Is that done by using a refresh token behind the scenes?

[JoelSpeed]

I doubt it is done using refresh tokens behind the scenes.

It's more likely that the cookie lasts a month and the act of clicking "login with google" just redirects you through to Google, but if you're signed in to google anyway, it just redirects you on again back to strava and you don't notice this little redirect loop.

I have a similar experience with the OAuth proxy in front of my Kubernetes but it doesn't use refresh tokens.

I don't know how to do it on browsers other than chrome, but with chrome you can open the developer tools and record the network traffic. Next time you click "login to google" you'll probably see it do a bunch of 302s before sending you back to the original page, a couple of these will likely be google transparently saying "yup, he's logged in"

[yveszoundi]

@JoelSpeed , are you able to share a recent build of your oauth2_proxy enhancements? I can't seem to be able to build the code locally (compilation errors). I'm also looking forward azure ad integration with oauth2_proxy. I've tested official builds against github and it looks like there are several issues with redirects, headers, etc.

[JoelSpeed]

@yveszoundi I've built an image from this branch and pushed it quay.io/joelspeed/oauth2_proxy:authorization-headers-1

[yveszoundi]

Thanks a lot @JoelSpeed, I'll have to learn myself some golang.

[chicco785]

@JoelSpeed if I get right, using this branch i would be able to use the oidc token to do the authentication agains the k8s api. any advice for k8s configuration?

thx

[chicco785]

ps: is really the image only 3.7 mb? I thought I did a good job with mine in 11.7mb... but yours is a master piece!

[JoelSpeed]

@chicco785 That's correct. This is only really for use in web browser flows though. You should set this proxy up in front of (or using auth urls in the nginx ingress controller) to provide bearer tokens as headers to the kube dashboard. For the raw API server you either need to extract this token from the header (you could have a webpage that people retrieve tokens from?) or have a separate authentication app that gets you this information. And as for image size, that's the compressed size, when extracted it's closer to the size of yours

[superbrothers]

This PR worked well for kubernetes dashboard with OIDC (coreos/dex). Thanks @JoelSpeed, Awesome PR!

[youngdev]

@JoelSpeed I can't seem to get this to work for dex and kubernetes dashboard. I am able to successfully authenticate through dex and land on the kubernetes dashboard page but do not see the bearer token passed along to the landing page and as a result unable to authenticate at this point. I do however see the bearer token in the response header of the auth2_proxy domain if I make a separate request to it. Below is my related settings:

        - --set-authorization-header=true
        - --pass-access-token=true
        - --pass-basic-auth=true

Do you happen have a build that includes both #464 and #534 ? Do you have a recommendation on which version of nginx-ingress-controller that is known to work with this implementation and any associated settings?

[JoelSpeed]

@youngdev Are you proxying the requests directly through the OAuth2 Proxy or are you using nginx with auth URls??

If you're running in nginx auth mode, try the following in your ingress annotations:

    ingress.kubernetes.io/auth-url: "https://oauth2.example.com/oauth2/auth"
    ingress.kubernetes.io/auth-signin: "https://oauth2.example.com/oauth2/start?rd=https://$host$request_uri$is_args$args"
    ingress.kubernetes.io/configuration-snippet: |
      # adds 301 redirect with trailing slash
      rewrite ^(/ui)$ $1/ permanent;
      # adds authorization header for kubernetes-dashboard
      auth_request_set $token $upstream_http_authorization;
      proxy_set_header Authorization $token;

And as for the image, yes I do: quay.io/joelspeed/oauth2_proxy:kubernetes-1 - this one contains both #464 and #534

[youngdev]

@JoelSpeed Awesome! Including the configuration-snippet annotation and config made it work for me. Thanks!

[chicco785]

hi @JoelSpeed all,

i am having the following issue, apparently, there is something in the configuration that does not work properly:

2018/03/23 16:27:48 [error] 252#252: *210 upstream sent too big header while reading response header from upstream, client: 213.55.184.136, server: auth.s.orchestracities.com, request: "GET /auth/realms/default/protocol/openid-connect/auth?approval_prompt=force&client_id=k8s-dashboard&redirect_uri=https%3A%2F%2Fkubernetes.s.orchestracities.com%2Foauth2%2Fcallback&response_type=code&scope=openid+email+profile&state=76dd1628d5eadadf65ca09c2cd68f25d%3A%2F HTTP/1.1", upstream: "http://100.103.64.6:8080/auth/realms/default/protocol/openid-connect/auth?approval_prompt=force&client_id=k8s-dashboard&redirect_uri=https%3A%2F%2Fkubernetes.s.orchestracities.com%2Foauth2%2Fcallback&response_type=code&scope=openid+email+profile&state=76dd1628d5eadadf65ca09c2cd68f25d%3A%2F", host: "auth.s.orchestracities.com", referrer: "https://auth.s.orchestracities.com/auth/realms/default/account/applications"

any idea?

thx!

[chicco785]

i tested using firefox and safari... it works... weird.

[chicco785]

k, so i enabled the buffer on nginx and increased it size, problem solved. still i am wondering why it was not working in chrome

[SEJeff]

Are there any code critiques causing this to not be merged? What's it waiting on?

[jhohertz]

@JoelSpeed : Could you share the Dockerfile used to build the containers mentioned above?

I am trying to test this w/ google as auth provider, and keep running into the following error, which I interpret to mean that root CA certs are not installed to the container (IE: package ca-certitificates if deb-based, and so on)

error redeeming code Post https://www.googleapis.com/oauth2/v3/token: x509: failed to load system roots and no roots provided

So I want to try creating a container w/ said CA certs.

Thanks!

[JoelSpeed]

@jhohertz The Dockerfile I've used is on this branch https://github.com/pusher/oauth2_proxy/blob/dockerfile/Dockerfile

For reference, we just mount /etc/ssl/certs/ca-certificates.crt from the host to solve this issue.

[jhohertz]

Thanks for the info, will try that approach.

[jhohertz]

Hi @JoelSpeed, i've just come back to this today, between now and last message I've implemented your variant of dex (with google refresh+groups) as our OIDC provider for kubernetes, and following your blog posts, trying to now get this working against that dex.

It's sort-of working, however by the time the dashboard gets the credentials, they no longer match the original user. IE: the same user who hits kubectl (and is seen as user oidc:<email> there), hits the dashboard and logs in, and dashboard reports permission errors for user oidc:<long hex string>. (And I assume groups are not passed, as they have no permissions, and the user in question does have said permissions when using CLI)

I am not 100% sure yet why this is happening. I am using the ingress-nginx approach with the annotations suggested here and on the blog page applied to the ingress for dashboard.

Any insights you might have regarding this would be much appreciated.

Thanks!

[tlawrie]

Hi @JoelSpeed my OIDC provider JWT is rather large due to my organizations addition of LDAP groups included in the JWT causing cookie size to be > 8kb

I am also using it in conjunction with Kubernetes NGINX ingress and am receiving
[error] 510#510: *8 upstream sent too big header while reading response header from upstream

Are you aware of any fix that can resolve this?

[ploxiln]

https://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers

[tlawrie]

@ploxiln thanks. I have already added this large_client_header_buffers 8 128k;

This worked for all headers up until the ones where the cookie being stored by Oauth2_proxy was > 8kb

[JoelSpeed]

@jhohertz Are you using separate clients for the CLI and Dashboard? Have you set up cross-client trust? https://github.com/coreos/dex/blob/master/Documentation/custom-scopes-claims-clients.md#cross-client-trust-and-authorized-party

If you check your OAuth proxy endpoint /oauth2/auth you should be able to see the JWT in the response headers. Decode the JWT. Check the aud field. To work with kubernetes, it needs to contain the value you set for --oidc-client-id on your API server definition?

I find the easiest way to debug these permissions issues is to use the dashboard to try and access a certain pod and then check the API server audit logs for what it thinks the user's identity is.

[tlawrie]

@JoelSpeed does Refresh Token work in this implementation? It seems when you enable refresh token, the NGINX is causing an error.

[MaksymBilenko]

@JoelSpeed Thank you for great work!
I have an issue with redirect on oauth2_proxy from your PR.

• Oauth proxy has no upstream
• Auth backend is dex
• Dex auth is GitHub

Almost the same configuration as you have in you blog post except that dex auth is GitHub instead of Google.

After successful auth process it goes back to auth.domain.com (oauth2_proxy) instead of dasboard.domain.com where I have correct rd parameter, token header and etc. at the end it goest to auth.example.com
And after if I go to dasboard it works fine.

Do you have any thoughts about this case?

[JoelSpeed]

@tlawrie I haven't experimented with refreshing within the oauth2_proxy yet. I know that it isn't in fact implemented in the OIDC connecter yet so it won't work at present. Our solution at the moment is that everyone has to refresh when their token expires.

I am planning to look at implementing the refreshing in the near future, but feel free to take this on yourself if it's something you need desperately.

@MaksymBilenko This kind of stuff is really hard to debug remotely. I'd suggest starting the authentication flow in an incognito window or equivalent and using the network tab of the browsers developer console to follow the requests through and see the redirect set properly and passed through.

Are you using my other PR as well? #464 as you may need that to enable the whitelist redirect as you require here

[jhohertz]

@JoelSpeed thanks for the feedback. I can confirm that the aud field of the JWT token matches the --oidc-client-id.

And in the process realized I can better describe what I am seeing.... it appears that the dashboard is seeing the "sub" field of the jwt token as the user ID, where we would expect to see the "email" field instead. I'm guessing this might have nothing to do with your changes, and more to do with how ingress-nginx is passing along the token. I am currently using this:

    nginx.ingress.kubernetes.io/configuration-snippet: |
      # adds authorization header for kubernetes-dashboard
      auth_request_set $token $upstream_http_authorization;
      proxy_set_header Authorization $token;

As for using separate clients, and cross-client trust, I am not 100% sure if this applies to this case? I mean yes, they are separate clients (dashboard accessed via chrome, kubectl via, well, kubectl...)

The weird thing is, if I disable the proxy, and take a token from the kubeconfig after logging in, and feed it to the dash, it works perfectly. I'll keep digging at it.

[Smana]

Hey @JoelSpeed great job,
Unfortunately, I didn't manage to get it working :
When i go to the kubernetes dashboard, i'm redirected to dex and the authentication works properly. However after that i got an http error 504. I'm pretty sure this is caused by my oauth2_proxy configuration.
Could you please provide the options you pass to oauth2_proxy ?

Mine looks like (I'm not sure what to put in upstream

      - args:
        - --provider=oidc
        - --http-address=0.0.0.0:8080
        - --client-id=auth-proxy
        - --client-secret=xxx
        - --redirect-url=https://auth.kube.domain.tld/oauth2/callback
        - --upstream=https://auth.kube.domain.tld
        - --oidc-issuer-url=https://dex.domain.tld/dex
        - --whitelist-domain=*
        - --cookie-domain=.domain.tld
        - --pass-basic-auth=false
        - --pass-access-token=false
        - --set-authorization-header=true
        - --cookie-secret=xxxx
        - --email-domain=*

The errors i can see are :

dashboard-proxy-58994cc79b-hws2k oauth-proxy 192.168.200.48 - smaine.kahlouch@domain.tld [27/May/2018:11:57:27 +0000] auth.kube.domain.tld GET auth.kube.domain.tld 192.168.200.48 - smaine.kahlouch@domain.tld [27/May/2018:11:57:27 +0000] auth.kube.domain.tld GET"/favicon.ico" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36" 502 0 0.003                      
dashboard-proxy-58994cc79b-hws2k oauth-proxy  auth.kube.domain.tld "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36" 502 0 0.002     
dashboard-proxy-58994cc79b-hws2k oauth-proxy 2018/05/27 11:57:27 reverseproxy.go:321: http: proxy error: context canceled                                                                                         
dashboard-proxy-58994cc79b-hws2k oauth-proxy 192.168.200.48 - smaine.kahlouch@domain.tld [27/May/2018:11:57:27 +0000] auth.kube.dm.gg GET auth.kube.domain.tld "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36" 502 0 0.001
dashboard-proxy-58994cc79b-hws2k oauth-proxy 192.168.200.48 - smaine.kahlouch@domain.tld [27/May/2018:11:57:27 +0000] auth.kube.domain.tld GET auth.kube.domain.tld "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36" 502 0 0.001
dashboard-proxy-58994cc79b-hws2k oauth-proxy 2018/05/27 11:57:27 reverseproxy.go:321: http: proxy error: context canceled                                                                                         
dashboard-proxy-58994cc79b-hws2k oauth-proxy 2018/05/27 11:57:27 reverseproxy.go:321: http: proxy error: context canceled                                                                                         
dashboard-proxy-58994cc79b-hws2k oauth-proxy 192.168.200.48 - smaine.kahlouch@dailymotion.com [27/May/2018:11:57:27 +0000] auth.kube.domain.tld GET auth.kube.domain.tld "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36" 502 0 0.000

Your help would be welcome :)

[charlesakalugwu]

@Smana upstream should be set to the service address of your kubernetes dashboard in the cluster. For example

--upstream=http://kubernetes-dashboard.kube-system.svc.cluster.local:9090

[bhack]

Will this work also with Github provider?

[Smana]

@charlesakalugwu thank you for your advice but how would that work when the oauth2_proxy is not located in the same kubernetes cluster ?

Actually i tried that because currently they're (kubernetes-dashboard and oauth2_proxy) running in the same cluster but after being authenticated i'm redirected to the login kubernetes-dashboard login page (the one giving the choice between the kubeconfig and a token).

To sum up what i'm trying to do:
1 - Open a browser to the kubernetes-dashboard ingress
config ingress controller :

    nginx.ingress.kubernetes.io/auth-url: "https://auth.kube.domain.tld/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://auth.kube.domain.tld/oauth2/start?rd=https://$host$request_uri$is_args$args" 
    nginx.ingress.kubernetes.io/configuration-snippet: |
      # adds authorization header for kubernetes-dashboard
      auth_request_set $token $upstream_http_authorization;
      proxy_set_header Authorization $token;

2 - Being redirected to Dex login page (LDAP backend)

3 - Being redirected to the kubernetes dashboard ingress with the authentication token

Maybe i'm missing something. Do I have to expose my kubernetes dashboard with a plain HTTP ingress ?

[jhohertz]

Just following up, this is working great for me now, main thing I was doing wrong was using --pass-authorization-header=true where I should have had --set-authorization-header=true for my use case.

👍 from me for getting this and the whitelist item merged.

[MaksymBilenko]

@JoelSpeed Thank you, it works!

[Smana]

@MaksymBilenko @jhohertz Could you please share the options you use with oauth_proxy ?
I didn't manage to get it work :/

[MaksymBilenko]

@Smana

1. I'm using this Dockerfile https://github.com/MaksymBilenko/oauth2_proxy_docker/blob/kubernetes/Dockerfile
2. with those args:

        - -provider=oidc
        - -redirect-url=https://auth.domain.com/oauth2/callback
        - -oidc-issuer-url=https://dex.domain.com
        - -cookie-secure=true
        - -cookie-expire=24h
        - -cookie-domain=.domain.com
        - -email-domain=*
        - -upstream=file:///dev/null
        - -set-authorization-header=true
        - -whitelist-domain=.domain.com
        - -skip-provider-button=true
        - -scope=openid profile email offline_access groups
        - -http-address=0.0.0.0:4180

[JoelSpeed]

That's very very similar to what we have set, but we don't actually set an upstream at all.

On another note, you should make sure the cookie domain and whitelist domains are the same here, although they don't necessarily need to have any relation to your issuer (We protect sites under x.kube.domain.com but our issuer is auth.domain.com so doesn't match the cookie or whitelist)

[Smana]

Thank you guys for your answers, unfortunately, I still have 502 errors. There must be something wrong with my setup. My ingress controller doesn't run in the kubernetes cluster. I'll dig further

[Smana]

After several tests, i finally got it working :)
However i'm wondering how to use it centrally because currently i have different client_ids (oidc) per clusters. I mean to get things work i would need to deploy a dashboard proxy per cluster :/

Here is an example of my dex config in order to have something that works

  - id: cluster-id-1
    name: "cluster-id-1"
    secret: "secret1"
    redirectURIs:
      - http://localhost:5555/callback
      - https://auth.domain.tld/oauth2/callback

  - id: cluster-id-2
    name: "cluster-id-2"
    secret: "secret2"
    redirectURIs:
      - http://localhost:5555/callback
      - https://auth.domain.tld/oauth2/callback

[JoelSpeed]

If anyone using this is using refresh tokens in other parts of their authentication flow, I've just implemented refreshing for the OIDC provider in #620 which may be of use to you

[JoelSpeed]

Closing this issue in favour of #621

Please see the new PR for the extra changes (refreshing in the background and proper validation)

[jcorioland]

Hi @Smana @JoelSpeed - I am struggling with securing the Kubernetes dashboard with the proxy, using the oidc provider. I am able to authenticate but then I receive a 502 error as you were. My configuration is similar to yours:

- args:
        - --provider=oidc
        - --oidc-issuer-url=https://login.microsoftonline.com/801dc5e8-3f2d-4070-ab27-a7497e430784/v2.0
        - --cookie-secure=true
        - --cookie-expire=24h
        - --email-domain=*
        - --upstream=file:///dev/null
        - --set-authorization-header=true
        - --whitelist-domain=.domain.com
        - --skip-provider-button=true
        - --scope=openid email
        - --http-address=0.0.0.0:4180

How did you manage to solve this issue?

Thanks!