Auth proxy does not work since version 6

[ug7n]

Tested versions:

Auth proxy working OK:
5.4.3

Auth proxy not working:
6.0.2
6.1.0-beta1
6.1.0-9a7577a1pre

grafana.ini:
[users]
allow_sign_up = true
allow_org_create = false
auto_assign_org = true
[auth.proxy]
enabled = true
header_name = X-WEBAUTH-USER
header_property = username
auto_sign_up = false
ldap_sync_ttl = 60

using nginx proxy, config:
proxy_set_header X-WEBAUTH-USER $user;
proxy_pass http://localhost:3000;

[torkelo]

Strange, in what way is it not working? Any errors?

[ug7n]

api work ok:
request:
curl -H “X-WEBAUTH-USER: user2” http://localhost:3000/api/user
answer:
{“id”:8,“email”:“user2@localhost”,“name”:"",“login”:“user2”,“theme”:"",“orgId”:1,“isGrafanaAdmin”:false}

Immediately after entering the page - logout.

There are no errors in the logs:
lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/api/dashboards/home status=401 remote_addr=XXX time_ms=0 size=26 referer=“XXX”
lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/api/login/ping status=401 remote_addr=XXX time_ms=0 size=26 referer=“XXX”
lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/logout status=302 remote_addr=XXX time_ms=0 size=29 referer=“XXX”

[juanmaz12]

We have been seen the exact same behaviour on our test deployment
When querying the API using curl, replies are as expected. However when accessing any grafana ver > 6 through the proxy existing users failed to authenthicate and non-existent users aren't created automatically.

[ug7n]

The problem is a bit more complicated.
Used auto login by url with nginx like:
https://github.com/grafana/grafana/issues/3752#issuecomment-423034369

nginx.conf:

worker_processes 1;
events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;

server {
    listen       80;
    server_name  localhost;
    location / {
            set $user "";
	set $state "";
	
	if ($args ~ "^user=(.+)&md5") {
                set $user $1;
                set $state "${state}U";
            }
	
	secure_link $arg_md5,$arg_expires;
            secure_link_md5 "$secure_link_expires$user for_test";
            
            if ($secure_link = "") { 
                set $state "${state}S1";
            }
            
            if ($secure_link = "0") { 
                set $state "${state}S2";
            }
            
            add_header X-uristate "$state";

            if ($state = "US1") { return 403; }
            if ($state = "US2") { return 410; }
            
            add_header X-uri "$user";
            proxy_set_header X-WEBAUTH-USER $user;
            proxy_pass http://localhost:3000;
    }
}

server {
    listen       81;
    server_name  localhost;
    location / {
            set $user "";
	set $state "";
	
	if ($args ~ "^user=(.+)&md5") {
                set $user $1;
                set $state "${state}U";
            }
	
	secure_link $arg_md5,$arg_expires;
            secure_link_md5 "$secure_link_expires$user for_test";
            
            if ($secure_link = "") { 
                set $state "${state}S1";
            }
            
            if ($secure_link = "0") { 
                set $state "${state}S2";
            }
            
            add_header X-uristate "$state";

            if ($state = "US1") { return 403; }
            if ($state = "US2") { return 410; }
            
            add_header X-uri "$user";
            proxy_set_header X-WEBAUTH-USER $user;
            proxy_pass http://localhost:3001;
    }
}

}

Configuration of grafana above,
80 -> 3000 port - version 5.4.3
81 -> 3001 port - version 6.1.3

Logs for 5.4.3:

request1:
GET http://localhost/?user=user1&md5=JnhztiDpOhrpKw59LRpIjQ&expires=2147483647 HTTP/1.1
Host: localhost
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
DNT: 1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,cy;q=0.6
Cookie: grafana_sess=400fba4395a72503; redirect_to=%252F

response1:

HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Mon, 15 Apr 2019 11:07:32 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
X-uristate: U
X-uri: user1
....

request2:

GET http://localhost/api/dashboards/home HTTP/1.1
Host: localhost
Connection: keep-alive
Accept: application/json, text/plain, /
DNT: 1
X-Grafana-Org-Id: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
Referer: http://localhost/?user=user1&md5=JnhztiDpOhrpKw59LRpIjQ&expires=2147483647
Accept-Encoding: gzip, deflate, br
Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,cy;q=0.6
Cookie: grafana_sess=400fba4395a72503; redirect_to=%252F

response2:
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Mon, 15 Apr 2019 11:07:32 GMT
Content-Type: application/json
Content-Length: 1491
Connection: keep-alive
Cache-Control: no-cache
Expires: -1
Pragma: no-cache
X-uristate: S1

{"meta":{"isHome":true,"canSave":false,"canEdit":false,"canAdmin":false,"canStar":false,"slug":"","url":"","expires":"0001-01-01T00:00:00Z","created":"0001-01-01T00:00:00Z","updated":"0001-01-01T00:00:00Z","updatedBy":"","createdBy":"","version":0,"hasAcl":false,"isFolder":false,"folderId":0,"folderTitle":"General","folderUrl":"","provisioned":false},"dashboard":{"annotations":{"list":[]},"editable":true,"folderId":null,"gnetId":null,"graphTooltip":0,"hideControls":true,"id":null,"links":[],"panels":[{"content":"\u003cdiv class="text-center dashboard-header"\u003e\n \u003cspan\u003eHome Dashboard\u003c/span\u003e\n\u003c/div\u003e","editable":true,"gridPos":{"h":3,"w":24,"x":0,"y":0},"id":1,"links":[],"mode":"html","style":{},"title":"","transparent":true,"type":"text"},{"folderId":0,"gridPos":{"h":17,"w":12,"x":0,"y":6},"headings":true,"id":3,"limit":4,"links":[],"query":"","recent":true,"search":false,"starred":true,"tags":[],"title":"","transparent":false,"type":"dashlist"},{"editable":true,"error":false,"gridPos":{"h":17,"w":12,"x":12,"y":6},"id":4,"links":[],"title":"","transparent":false,"type":"pluginlist"}],"rows":[],"schemaVersion":16,"style":"dark","tags":[],"templating":{"list":[]},"time":{"from":"now-6h","to":"now"},"timepicker":{"hidden":true,"refresh_intervals":["5s","10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options":["5m","15m","1h","6h","12h","24h","2d","7d","30d"],"type":"timepicker"},"timezone":"browser","title":"Home","version":0}}

Everything works fine.

Logs for 6.1.3:

request1:
GET http://localhost:81/?user=user1&md5=JnhztiDpOhrpKw59LRpIjQ&expires=2147483647 HTTP/1.1
Host: localhost:81
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
DNT: 1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,cy;q=0.6
Cookie: redirect_to=%252F

response1:
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Mon, 15 Apr 2019 11:09:28 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: grafana_sess=155821a37ae7e80a; Path=/; HttpOnly
X-uristate: U
X-uri: user1
....

request2:

GET http://localhost:81/api/dashboards/home HTTP/1.1
Host: localhost:81
Connection: keep-alive
Accept: application/json, text/plain, /
DNT: 1
X-Grafana-Org-Id: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
Referer: http://localhost:81/?user=user1&md5=JnhztiDpOhrpKw59LRpIjQ&expires=2147483647
Accept-Encoding: gzip, deflate, br
Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,cy;q=0.6
Cookie: redirect_to=%252F; grafana_sess=155821a37ae7e80a

response2:
HTTP/1.1 401 Unauthorized
Server: nginx/1.14.2
Date: Mon, 15 Apr 2019 11:09:29 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 26
Connection: keep-alive
Cache-Control: no-cache
Expires: -1
Pragma: no-cache

{"message":"Unauthorized"}

[juanmaz12]

I think we can provide a clearer explanation of the bug.

It all starts when the browser requests something like http(s)://${proxy_domain}/${proxy_path}/d/${dashboard_id}/${dashboard_name}?${some_parameters}.
The reverse proxy transforms the request into http(s)://${grafana_domain}/d/${dashboard_id}/${dashboard_name}?${some_parameters}, adds an x-webauth-user header and sends the modified request to the grafana server.

Our expectation, and the behaviour observed in previous versions of grafana, is that in response to such request, grafana replies with appropriate content, but including in the response headers a cookie containing some grafana session id.
After that, the browser includes the cookie in all subsequent requests, grafana server code considers the user logged-in because of the cookie and replies with appropriate content, and the reverse proxy does not need to include the x-webauth-header in every subsequent request.

Now in recent versions of grafana that behaviuor changed, or broke, if you prefer. The response to the initial request (the request containing the x-webauth-user header that loads the grafana client code in the browser) does not include a cookie with the session id in the headers on grafana versions greater than 6.x.

The cookie with the session id eventualy comes backs to the browser after a few requests that the grafana client code sends to the grafana API, such as /api/dashboards/uid/${dashboard_id} and /api/login/ping.
But unless those API requests are sent with the x-authuser-request the API rejects them (rightly so) and the grafana client code redirects the user to the usual interactive login page.

So, if the reverse proxy includes the x-webauth-user header in every request, the proxy authentication works.
But if the reverse proxy only includes the x-webauth-user header in the initial request (the request that loads the grafana client code in the browser) then the proxy authentication fails to work.

In our application, the initial request includes in the query string a parameter that is inspected by our reverse proxy code to authorize the request. Our reverse proxy code only adds the x-webauth-user header if the query string parameter includes said additional parameter and it has a valid value. But of course the grafana client code is unaware of our reverse proxy authorization scheme and will not include our additional query string parameter in requests it sends to the grafana server.

Therefore it is important for us that the initial request (the request that loads the grafana client code in the browser) be replied by the grafana server including the session id cookie needed to authorize subsequent requests.

[mryiiiii]

Thanks @juanmaz12 for this explanation. I just spent hours trying to figure out the root cause for this regression.
Hope a quick fix can happen soon.

Thanks!

Yi

[torkelo]

That is very interesting, great investigation. That the auth proxy worked before without having auth proxy headers included in every request is new to us and definitely a bug , auth proxy is intended to proxy all requests.

[Bitblade]

We use auth proxy in exactly the same way as @juanmaz12 and are having the same problems.
@juanmaz12 great work figuring out what is going on!

In fact, I believe several examples for autologin (Status screens hanging on the wall) in the issues here show this exact use case.

As a workaround, I managed to get it working again by setting a cookie and testing for the cookies existence.

I used this addition to my apache configuration. Requires Apache2.4 or higher.
(Please note: this is a very quick proof of concept. It's quick and dirty. Also, it is not secure in any way as I don't need any serious security for this setup.)

<If "%{QUERY_STRING} =~ m#&?wall=<KEY HERE>#">
  RequestHeader set X-WEBAUTH-USER <GRAFANA USER HERE>
  RewriteEngine On
  RewriteRule .* -  [CO=wall:<KEY HERE>:<FQDN HERE>:0:/:true:true] # Sets cookie 'wall'
</If>
<If "%{HTTP_COOKIE} =~ /wall=<KEY HERE>/">
  RequestHeader set X-WEBAUTH-USER <GRAFANA USER HERE>
</If>

[provenvelocity]

#11408

Another way of seeing the issue...

[Marc3001]

We got same issue here, we are using link with token to rewrite grafana url with auth_proxy user header.
Once this url used, we were expecting grafana_session cookie to be part of grafana response and allow user to navigate UI like he was logged-in normally.
Here with 6.X versions, we are not able to make it work this way as if grafana is not pushing session cookie, we should propagate token through all calls performed by UI, and as far as I know, this is not possible through grafana conf.

[advaitvdeo]

We are facing same issue where /api/ call is giving status=400 when we are trying to save existing dashboard.
We are using auth.proxy to pass the header

Could you please suggest what could be wrong in our case?

[torkelo]

are you always passing the auth proxy header?

[advaitvdeo]

Yes. We are passing auth proxy header always. Still when we try to save dashboard its giving status=400

[Marc3001]

@advaitvdeo HTTP400 means some of params (not headers) you're passing are missing or malformed. Check message in body to get more info about which param.
This does not seem to be related to auth.proxy

[gzzo]

If it helps the grafana folks in here, a lot of people are using a variation of this #3752 (comment) for their reverse proxy setup.

Basically, we only want to include the auth proxy header (X-WEBAUTH-USER) if the request provides some authentication information. For the example above, the initial request to grafana is signed with some query parameters and the reverse proxy only includes the header if the signature matches the request.

Since the API calls that follow the initial load don't include those parameters, and grafana 6.x does not behave the same with cookies as 5.x did and (as @juanmaz12 pointed out) access is denied.

[torkelo]

Created a new feature request to track the interest in a feature to mix auth proxy and Grafana's login tokens,. #17316

[SvenMuc]

Any workaround for this issue? Have the same trouble with Grafana v6.3.2

[garyyang85]

So is there a plan when will fix this issue? Have to retrun to 5.X version now....

[torkelo]

Just have the auth proxy proxy all urls not just login

[torkelo]

But this is fixed in 6.5 or 6.6 ,
We added a new setting to have auth proxy generate login tokens when accessed via login page.

[garyyang85]

But this is fixed in 6.5 or 6.6 ,
We added a new setting to have auth proxy generate login tokens when accessed via login page.

@torkelo same configuration works on 5.X version， but doesn't work on 7.0.4. So this may be a regression bug?
URL: http://localhost:8080/grafana/?user=testuser1
Nginx configuration:
location /grafana/ {
proxy_set_header Authorization “”;
proxy_set_header Host $host;
proxy_set_header X-WEBAUTH-USER $arg_user;
proxy_pass http://localhost:3000/;
proxy_redirect off;
}