Authorization header is passed to upstream endpoint if it is passed on the client side though enable-authorization-header=false

[sandeepbangera]

Title

Authorization header is passed to upstream endpoint if it is passed on the client side e.g. using curl though enable-authorization-header flag is set to false. This causes issues for upstream endpoints like grafana which act on the authorization header if passed.

Summary

Tried accessing grafana API endpoint which is front ended by louketo proxy and API access FAILS because the proxy passes the Authorization header to the upstream grafana endpoint.

Environment

• OS: Mac
• Kernel: Darwin 19.5.0 Darwin Kernel Version 19.5.0:
• Go: go version go1.13.7 darwin/amd64
• Server : keycloak
• Louketo: 1.0.0

Expected Results

Expected the curl command to invoke the Grafana API to work.

Actual Results

Grafana API access fails with "Invalid API key"

Steps to reproduce

I was trying to use the gatekeeper proxy quay.io/louketo/louketo-proxy:1.0.0 as a side car inside container to talk to Grafana. Grafana has been configured to run in Auth Proxy mode.

  auth.proxy:
    enabled: true
    header_name: X-Auth-Username
    header_property: username
    auto_sign_up: true
    ldap_sync_ttl: 10

Verified that the auth proxy configuration with vanilla grafana works fine.
The gatekeeper is now inserted as a side car inside the grafana container with the following args

    --discovery-url=https://keycloak.mysample.com/auth/realms/myrealm
    --client-id=some-client
    --client-secret=some-secret
    --upstream-url=http://127.0.0.1:3000
    --enable-default-deny=true
    --listen=0.0.0.0:9082
    --resources=uri=/*
    --scopes=good-service
    --enable-authorization-header=false
    --verbose=true
    --enable-logging=true
    --secure-cookie=false
    --enable-refresh-tokens=true
    --encryption-key=somekey

The gatekeeper intercepts the call, does the authentication with the keycloak server and if the authentication is successful forwards the request to Grafana.
This works perfectly from the browser. But trying to call the grafana API from command line FAILS.

MY_ADMIN_TKN=$(curl -XPOST -s https://keycloak.mydomain.com/auth/realms/myrealm/protocol/openid-connect/token \
                           -H "Content-Type: application/x-www-form-urlencoded" \
                           -d 'username=myadminuser' \
                           -d 'password=mypasswd' \
                           -d 'grant_type=password' \
                           -d 'client_id=some-client' \
                           -d 'client_secret=mysecret' \
                           -d 'scope=openid' | jq -r '.access_token')

curl -XGET -s https://grafana.mydomain.com/api/org \
           -H 'Accept: application/json' -H 'Content-Type: application/json' -H "Authorization: Bearer $MY_ADMIN_TKN"

{"message":"Invalid API key"}

From the louketo proxy logs the authentication was successful and the proxy is passing the Authorization header to the upstream endpoint Grafana. Grafana rejects the request because it cannot recognize the authorization header passed. If you see my snippet above I do have --enable-authorization-header=false passed to the proxy.

Additional Information

Looking at the code we pass through all the headers we get from the original client i.e. curl in this case to the upstream endpoint.

I was able to get the correct behavior by adding the below code to drop the authorization header here:
https://github.com/louketo/louketo-proxy/blob/master/middleware.go#L513

			if !r.config.EnableAuthorizationHeader && req.Header.Get("Authorization") != "" {
				req.Header.Del("Authorization")
			}

I will be happy to create a pull request to fix this.

[vasilievs]

if you use oauth on proxy, why need to use grafana oauth setting?

Role-base model want to use in gafana?

yesterday i setup grafana with louketo (with use custom port to grafana backend)

    - --enable-authorization-header=false
    - --preserve-host=true

and grafana:

[auth.anonymous]
# enable anonymous access
enabled = true
...
# specify role for unauthenticated users
org_role = Admin

[auth.basic]
enabled = false

[jacky96623]

if you use oauth on proxy, why need to use grafana oauth setting?

Role-base model want to use in gafana?

yesterday i setup grafana with louketo (with use custom port to grafana backend)

    - --enable-authorization-header=false
    - --preserve-host=true

and grafana:

[auth.anonymous]
# enable anonymous access
enabled = true
...
# specify role for unauthenticated users
org_role = Admin

[auth.basic]
enabled = false

I think the focus should be on failed enable-authorization-header config but not Grafana?
This issue occurs for any proxy usage e.g. httpbin, or even if you just run a Gatekeeper without a healthy upstream.

[vasilievs]

once again revised my config, I set
authorization header false,
- --enable-authorization-header=false
removing it got an error in grafana
{"message":"Invalid API key"}

my version - docker-compose 2.4,
image proxy - quay.io/louketo/louketo-proxy
image grafana - grafana/grafana:7.0.6

[sandeepbangera]

@vasilievs , I did debug the louketo code (Printed everything it passed to the upstream endpoint), It does pass the authorization header to grafana even if --enable-authorization-header=false .. I recommended a fix in the code in my original post which fixes the problem. I rebuilt the container locally with the fix and was able to verify that I no longer get {"message":"Invalid API key"} when I call the grafana REST API using curl and my rebuilt local louketo proxy container.

@jacky96623 I apologize but I don't understand your point. The basic problem is that Grafana REST API calls are failing if Grafana is configured in auth proxy mode with Louketo proxy in front of grafana and we pass --enable-authorization-header=false flag to the proxy container (With this flag set louketo proxy should not be passing the authorization header to the upstream endpoint in this case grafana). Everything works fine from the browser. This behavior is very specific to Grafana. When Grafana is in auth proxy mode, if you pass it the Authorization header it still tries to read it and throws the {"message":"Invalid API key"} . In theory Grafana also could fix this on their end that they should ignore the authorization header when it runs in Auth proxy mode. The whole idea of Auth Proxy is that Grafana is delegating the Authentication to a different IDP.

[jacky96623]

@sandeepbangera First of all let me apologize for not expressing my opinion clear.

Indeed I face the similar problem as you, the only difference is that I am not proxying for Grafana. And my steps to reproduce is just start a gatekeeper which proxies httpbin.

The detailed configuration are as follow:

docker run -p 3000:3000 \
  quay.io/keycloak/keycloak-gatekeeper:10.0.0 \
  --listen=:3000 \
  --discovery-url=http://keycloak:8080/auth/realms/my-realm \
  --client-id=my-client-id \
  --client-secret=my-client-secret \
  --upstream-url=https://httpbin.org \
  --enable-authorization-header=false

With the above configuration and a valid token, I can get my Authorization header back by simply accessing http://localhost:3000/get

But if you think Grafana's auth proxy mode should handle this issue, maybe you should also raise it to Grafana?

[vasilievs]

the key --enable-authorization-header=false doesn't seem to work as it should
on kibana i see this behavior with parameters, in the response I see all headers from the proxy:

 "headers":{
         "host":"kibana:5602",
         "connection":"close",
         "user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36",
         "accept":"*/*",
         "accept-language":"en-US,en;q=0.9",
         "referer":"https://host:5601/app/kibana",
         "sec-fetch-dest":"script",
         "sec-fetch-mode":"no-cors",
         "sec-fetch-site":"same-origin",
         "x-auth-audience":"louketo-client",
         "x-auth-email":"monitoring@email.ru",
         "x-auth-expiresin":"2020-08-20 13:02:53 +0000 UTC",
         "x-auth-groups":"",
         "x-auth-roles":"admin",
         "x-auth-subject":"abf97d2a-c344-41ac-b639-c9434705ceab",
         "x-auth-userid":"monitoring",
         "x-auth-username":"monitoring",
         "x-forwarded-for":"192.168.13.45",
         "x-forwarded-host":"host:5601",
         "x-forwarded-proto":"",
         "accept-encoding":"gzip"
      },

keys set in docker-compose (i tried with all variants (one or some or all, only the token-header is removed correctly)):

    - --enable-session-cookies=false
    - --enable-token-header=false
    - --enable-authorization-header=false
    - --enable-authorization-cookies=false