Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grafana auth proxy scenario not working for dashboard calls #266

Closed
m-wcislo opened this issue Sep 26, 2019 · 6 comments
Closed

Grafana auth proxy scenario not working for dashboard calls #266

m-wcislo opened this issue Sep 26, 2019 · 6 comments
Labels

Comments

@m-wcislo
Copy link

I'm using oauth2_proxy to protect Grafana. Basic flows are working, I'm redirected to IdP (Keycloak) and then logged in as my user. I have a problem with dashboards which are triggering requests towards /api/ and those are failing after token/cookie is expired. After refresh it starts working again for the time of token expiration.

Expected Behavior

All the browser side requests should follow OIDC flow and be authenticated as long as session on Keycloak side is valid. Tokens should be refreshed.

Current Behavior

Grafana UI requests only works as long as token is valid and it is not refreshed.

Possible Solution

Check out logs/flow below. The thing that seems to be not ok (although not sure if this is the whole problem) is that request path is passed as state:

&state=baf621047a3bf084685162adcdf2db9a%3A%2Fpublic%2Fimg%2Ficons_dark_theme%2Ficon_dashboard_list.svg

while

set-cookie: _oauth2_proxy_csrf=baf621047a3bf084685162adcdf2db9a

by the way I'm using

nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$escaped_request_uri

because Grafana is passing query parameters there and I guess it could break things having extra parameters in OIDC flow which might be something to be fixed too.

Steps to Reproduce (for bugs)

  1. Access grafana...., follow OIDC flow
  2. wait for 2 min, for token to expire
  3. Do listing dashboards, or observe dashboard. Token refresh happens, and from UI perspective works fine.
    Browser:
https://grafana.michu2.someuser.cluster.company.net/public/img/icons_dark_theme/icon_dashboard_list.svg

https://grafana.michu2.someuser.cluster.company.net/oauth2/start?rd=%2Fpublic%2Fimg%2Ficons_dark_theme%2Ficon_dashboard_list.svg

>>
    set-cookie: _oauth2_proxy_csrf=baf621047a3bf084685162adcdf2db9a; Path=/; Expires=Thu, 03 Oct 2019 09:47:53 GMT; Secure
    status: 302

https://keycloak.michu2.someuser.cluster.company.net/auth/realms/master/protocol/openid-connect/auth?approval_prompt=force&client_id=sso&redirect_uri=https%3A%2F%2Fgrafana.michu2.someuser.cluster.company.net%2Foauth2%2Fcallback&response_type=code&scope=openid+email+profile&state=baf621047a3bf084685162adcdf2db9a%3A%2Fpublic%2Fimg%2Ficons_dark_theme%2Ficon_dashboard_list.svg

https://grafana.michu2.someuser.cluster.company.net/oauth2/callback?state=baf621047a3bf084685162adcdf2db9a%3A%2Fpublic%2Fimg%2Ficons_dark_theme%2Ficon_dashboard_list.svg&session_state=4d73f40b-2fca-4467-82aa-958beca58dda&code=5c64d682-066e-4534-8cab-8dd6ebbefd5d.4d73f40b-2fca-4467-82aa-958beca58dda.13fc2b11-f57e-41e1-9553-41ef90ef7b57

>>Response
    set-cookie: _oauth2_proxy_csrf=; Path=/; Expires=Thu, 26 Sep 2019 08:47:53 GMT; Secure
    set-cookie: _oauth2_proxy_0=eyJ...

Oauth2_proxy logs:

[2019/09/26 09:47:52] [oauthproxy.go:750] Refreshing 2m17.88056502s old session cookie for Session{email:michal.someuser@company.com user:someuser token:true id_token:true created:2019-09-26 09:45:34.11943498 +0000 UTC expires:2019-09-26 09:46:34 +0000 UTC refresh_token:true} (refresh after 30s) 
(...)
[2019/09/26 09:47:52] [oauthproxy.go:745] Error loading cookied session: Cookie Signature not valid 
10.156.55.210 - - [2019/09/26 09:47:52] grafana.michu2.someuser.cluster.company.net GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" 401 21 0.001 
10.156.55.210 - - [2019/09/26 09:47:53] grafana.michu2.someuser.cluster.company.net GET - "/oauth2/start?rd=%2Fpublic%2Fimg%2Ficons_dark_theme%2Ficon_dashboard_list.svg" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" 302 452 0.000 
10.156.55.210 - michal.someuser@company.com [2019/09/26 09:47:53] [AuthSuccess] Authenticated via OAuth2: Session{email:michal.someuser@company.com user:someuser token:true id_token:true created:2019-09-26 09:47:53.329583572 +0000 UTC m=+516.498996143 expires:2019-09-26 09:48:53 +0000 UTC refresh_token:true} 
10.156.55.210 - - [2019/09/26 09:47:53] grafana.michu2.someuser.cluster.company.net GET - "/oauth2/callback?state=baf621047a3bf084685162adcdf2db9a%3A%2Fpublic%2Fimg%2Ficons_dark_theme%2Ficon_dashboard_list.svg&session_state=4d73f40b-2fca-4467-82aa-958beca58dda&code=5c64d682-066e-4534-8cab-8dd6ebbefd5d.4d73f40b-2fca-4467-82aa-958beca58dda.13fc2b11-f57e-41e1-9553-41ef90ef7b57" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" 302 75 0.147 
10.156.55.210 - michal.someuser@company.com [2019/09/26 09:47:53] grafana.michu2.someuser.cluster.company.net GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" 202 0 0.000 
[2019/09/26 09:47:53] [oauthproxy.go:631] Error redeeming code during OAuth2 callback: token exchange: oauth2: cannot fetch token: 400 Bad Request 
Response: {"error":"invalid_grant","error_description":"Code not valid"} 
  1. Wait whatever, seems the cookie is already expired anyway
    Browser:
https://grafana.michu2.someuser.cluster.company.net/api/search?dashboardIds=9&dashboardIds=3&dashboardIds=8&dashboardIds=1

https://grafana.michu2.someuser.cluster.company.net/oauth2/start?rd=%2Fapi%2Fsearch%3FdashboardIds%3D9%26dashboardIds%3D3%26dashboardIds%3D8%26dashboardIds%3D1

>>
    set-cookie: _oauth2_proxy_csrf=6c5058b1e7f91124eb846e504f9b7aff; Path=/; Expires=Thu, 03 Oct 2019 09:50:45 GMT; Secure
    status: 302

https://keycloak.michu2.someuser.cluster.company.net/auth/realms/master/protocol/openid-connect/auth?approval_prompt=force&client_id=sso&redirect_uri=https%3A%2F%2Fgrafana.michu2.someuser.cluster.company.net%2Foauth2%2Fcallback&response_type=code&scope=openid+email+profile&state=6c5058b1e7f91124eb846e504f9b7aff%3A%2Fapi%2Fsearch%3FdashboardIds%3D9%26dashboardIds%3D3%26dashboardIds%3D8%26dashboardIds%3D1

Oauth2_proxy logs:

(...)
[2019/09/26 09:50:45] [oauthproxy.go:750] Refreshing 2m51.670416428s old session cookie for Session{email:michal.someuser@company.com user:someuser token:true id_token:true created:2019-09-26 09:47:53.329583572 +0000 UTC expires:2019-09-26 09:48:53 +0000 UTC refresh_token:true} (refresh after 30s) 
(...)
[2019/09/26 09:50:45] [oauthproxy.go:755] 10.42.7.0:60990 ("10.156.55.210") removing session. error refreshing access token unable to redeem refresh token: failed to get token: oauth2: cannot fetch token: 400 Bad Request 
Response: {"error":"invalid_grant","error_description":"Session doesn't have required client"} Session{email:michal.someuser@company.com user:someuser token:true id_token:true created:2019-09-26 09:47:53.329583572 +0000 UTC expires:2019-09-26 09:48:53 +0000 UTC refresh_token:true}

Context

Grafana auth proxy configuration on kubernetes cluster. I'm using ingress controller auth module to do redirects to oauth2_proxy.

Your Environment

kubernetes cluster, with nginx-ingress-controller:0.21.0-

  • Version used: 4.0.0
    oauth2_proxy.cfg:
    upstreams = [ "file:///dev/null" ]
    cookie_expire = "168h"
    cookie_refresh = "30s"
    cookie_httponly = false
    cookie_secure = false

tried setting cookie_domain, but it didn't helped.

Seem to be similar to #29 but I'm getting set-cookie headers so I guess it is not it, but the flow of problem is the same.

@tlex
Copy link

tlex commented Oct 16, 2019

I can confirm this happening with 4.0.0, but not with 3.2.0 - using GitLab (self hosted), not Keycloak. It seems that the token refresh doesn't work.

Edit: I use the native oauth implementation in Grafana, but I've had the problem with graylog, karma, nzbget. Basically, anything doing ajax requests.

@m-wcislo
Copy link
Author

I'm using here cookie store, which turned out to have problems with sending back _oauth2_proxy_x cookies in my setup (NGINX in auth request mode). From another thread, the issue goes:

1. access endpoint for the first time, both _auth_proxy_0 and _auth_proxy_1 are send to client at /callback step
2. do browser refresh, this triggers cookie refresh, but only _auth_proxy_0 is send to client
3. Getting any resource from endpoint (like favicon.ico) fails due to corrupted cookie - cookie consists of _auth_proxy_0 and "old" _auth_proxy_1, whole authorization code flow is triggered
4. Again at /callback step cookie is set properly, page loads without relogin

So with working cookies the auth flow should never happen, because oauth2_proxy is replacing token silently. I just tested it with redis store and it works. So the original issue exists but with redis you should not see it until you are able to use refresh token. Can you confirm that you are using cookie store too?

For redis testing I used:
helm install stable/redis --set usePassword=false

and add following to oauth2 settings:

  redis-connection-url: "redis://redis-master.<namespace>.svc.cluster.local:6379"
  session-store-type: "redis"

@tlex
Copy link

tlex commented Oct 17, 2019

I'm using docker swarm with containous/traefik in front of oauth2_proxy. No credential store. This is the relevant part of the compose file:

version: "3.7"

services:
   oauth:
    # image: quay.io/pusher/oauth2_proxy:v3.2.0
    image: quay.io/pusher/oauth2_proxy:latest
    networks:
      - oauth
      - oauth-web
    deploy:
      restart_policy:
        delay: 5s
      labels:
        ai.ix.auto-update: 'true'
        ai.ix.expose: 'true'
        ai.ix.fqdn: ${FQDN_OAUTH?err}
        traefik.enable: 'true'
        traefik.docker.network: oauth-web
        traefik.http.routers.__STACK_NAME__-http.middlewares: default-http@file
        traefik.http.routers.__STACK_NAME__-http.entrypoints: http
        traefik.http.routers.__STACK_NAME__.entrypoints: https
        traefik.http.routers.__STACK_NAME__.middlewares: default-https@file
        traefik.http.routers.__STACK_NAME__.tls.certResolver: 'default'
        traefik.http.routers.__STACK_NAME__.service: __STACK_NAME__@docker
        traefik.http.services.__STACK_NAME__.loadbalancer.server.port: '4180'
    environment:
      OAUTH2_PROXY_CLIENT_ID: '${CLIENT_ID?err}'
      OAUTH2_PROXY_CLIENT_SECRET: '${CLIENT_SECRET?err}'
      OAUTH2_PROXY_COOKIE_SECRET: '${COOKIE_SECRET?err}'
      OAUTH2_PROXY_HTTP_ADDRESS: '0.0.0.0:4180'
      OAUTH2_PROXY_COOKIE_DOMAIN: '${UPSTREAM_COOKIE_DOMAIN?err}'
      OAUTH2_PROXY_COOKIE_SECURE: 'true'
      OAUTH2_PROXY_COOKIE_REFRESH: '10m'
      OAUTH2_PROXY_EMAIL_DOMAINS: '*'
      OAUTH2_PROXY_PASS_BASIC_AUTH: 'false'
      OAUTH2_PROXY_PASS_USER_HEADERS: 'true'
      OAUTH2_PROXY_SET_AUTHORIZATION_HEADER: 'true'
      OAUTH2_PROXY_SET_XAUTHREQUEST: 'true'
      OAUTH2_PROXY_FOOTER: '-'
      OAUTH2_PROXY_PROVIDER: 'gitlab'
      # OAUTH2_PROXY_LOGIN_URL: 'https://${FQDN_GIT?err}/oauth/authorize'
      # OAUTH2_PROXY_REDEEM_URL: 'https://${FQDN_GIT?err}/oauth/token'
      # OAUTH2_PROXY_VALIDATE_URL: 'https://${FQDN_GIT?err}/api/v4/user'
      # OAUTH2_PROXY_REDIRECT_URL: 'https://${FQDN_OAUTH?err}'
      OAUTH2_PROXY_OIDC_ISSUER_URL: 'https://${FQDN_GIT?err}'
    command:
      - -upstream=http://${UPSTREAM_SERVICE?err}:${UPSTREAM_PORT?err}
networks:
  oauth-web:
    external: true
  oauth:
    driver: overlay
    driver_opts:
      encrypted: 'true'
    internal: true

I've already tried with different OAUTH2_PROXY_COOKIE_REFRESH times. The commented out parts are what I use (successfully) with oauth2_proxy v3.2.0

@m-wcislo
Copy link
Author

By default oauth2_proxy uses cookie store, which is your case too. You need to switch to redis and you should not see this problem (see https://pusher.github.io/oauth2_proxy/configuration/sessions). But I still think there is some problem with state setting.

@tlex
Copy link

tlex commented Oct 17, 2019

From the log:

[2019/10/17 19:34:51] [oauthproxy.go:758] 10.0.79.5:55934 ("YY.XXX.NNN.NNN") removing session. error refreshing access token unable to redeem refresh token: failed to get token: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_request","error_description":"The request is missing a required parameter, includes an unsupported parameter value, or is otherwise malformed."} Session{email:YYY@XXXX.XX user:XXXX token:true id_token:true created:2019-10-17 19:32:48.387109319 +0000 UTC expires:2019-10-17 19:34:48 +0000 UTC refresh_token:true}

Now, this was tested with:

    command:
      - --upstream=http://${UPSTREAM_SERVICE?err}:${UPSTREAM_PORT?err}
      - --session-store-type=redis
      - --redis-connection-url=redis://tasks.redis_redis:6379

The above error appeared exactly 2 minutes in.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2020

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants