unauthorized errors when using reverse proxy with sub-path method

[jeroenjacobs79]

I'm using a nginx reverse-proxy with the sub-path method.

I'm getting 401 errors on xhr requests quite regularly. Refreshing the page sometimes solves the issue, sometimes not.

I'm also unable to sign out. When I click signout, I 'm redirect to the grafana main screen again. Trying it again a few times, just results in a "too many redirects" after a while. The only way I can sign out is by clearing the grafana cookies.

As for I know, these issues only pop up when using sub-path reverse proxy. Not sure if TLS termination on the reverse proxy has an impact, haven't tested that.

I'm using Grafana 5.0.4

My nginx.conf:

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

server {
        listen 443 ssl default_server;
server_tokens off;
ssl_certificate /etc/letsencrypt/live/home.mydomain.be/fullchain.pem ;
ssl_certificate_key /etc/letsencrypt/live/home.mydomain.be/privkey.pem ;
ssl_protocols TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_prefer_server_ciphers on;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
ssl_trusted_certificate /etc/letsencrypt/live/home.mydomain.be/chain.pem;

add_header Strict-Transport-Security max-age=15768000;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Xss-Protection "1; mode=block" always;
# add_header Referrer-Policy "strict-origin-when-cross-origin" always;
# add_header Content-Security-Policy "default-src 'self'"


        server_name home.mydomain.be;

proxy_buffering off;

location /grafana/ {
proxy_pass http://localhost:3000/;
# proxy_set_header Host $host;
# proxy_redirect http:// https://;
# proxy_http_version 1.1;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection $connection_upgrade;
}

}

server {
        listen 80 default_server;
        listen [::]:80 default_server;
        server_name home.mydomain.be;
        return 301 https://$server_name$request_uri;
}

relevant section of grafana.ini:

[server]
# Protocol (http, https, socket)
protocol = http

# The ip address to bind to, empty will bind to all interfaces
http_addr = 127.0.0.1

# The http port to use
;http_port = 3000

# The public facing domain name used to access grafana from a browser
domain = home.mydomain.be

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = https://%(domain)s:/grafana

I already commented out some security header stuff(Content-Security-Policy, Referer-Policy) in my nginx.conf, as I assumed it might cause the issue. However, this doesn't seem to make a difference.

[daniellee]

Don't know if this helps but there is a comprehensive guide to nginx in the digital oceans tutorials:

https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-grafana-on-ubuntu-16-04

We have a very simple nginx config for testing: https://github.com/grafana/grafana/blob/master/docker/blocks/nginx_proxy/nginx.conf

I'm not seeing the unauthorized problems but I am seeing the sign out problem. Will investigate more and get back to you.

[xrstf]

I am experiencing similar issues. Since upgrading Grafana to 5.1.x I'm seeing these unauthorized errors quite regularly. The server logs indicate a "user not found" problem:

t=2018-05-26T10:18:43+0000 lvl=eror msg="Failed to get user with id" logger=context userId=1 error="User not found"
t=2018-05-26T10:18:43+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/datasources/proxy/1/query status=401 remote_addr=1.2.3.4 time_ms=3 size=26 referer="https://...."
t=2018-05-26T10:18:43+0000 lvl=info msg="Database table locked, sleeping then retrying" logger=sqlstore retry=0
t=2018-05-26T10:18:43+0000 lvl=info msg="Database table locked, sleeping then retrying" logger=sqlstore retry=0
t=2018-05-26T10:18:43+0000 lvl=info msg="Database table locked, sleeping then retrying" logger=sqlstore retry=0
t=2018-05-26T10:18:43+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/login/ping status=401 remote_addr=1.2.3.4 time_ms=44 size=26 referer="https://...."

The errors do not occur when I have the Firefox developer tools opened, for some reason.

I'm running behind an nginx reverse proxy that's setup like this:

server {
  listen 80;
  listen [::]:80;

  server_name grafana.example.com;

  access_log /var/log/nginx/grafana-access.log;
  error_log /var/log/nginx/grafana-error.log;

  # include Let's Encrypt redirect
  include snippets/letsencrypt.conf;

  location / {
    return 307 https://$host$request_uri;
  }
}

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;

  server_name grafana.example.com;

  access_log /var/log/nginx/grafana-access.log;
  error_log /var/log/nginx/grafana-error.log;

  # SSL configuration
  ssl_certificate /ssl/acme/live/grafana.example.com/fullchain;
  ssl_certificate_key /ssl/acme/live/grafana.example.com/privkey;

  # improve SSL
  include snippets/ssl.conf;

  # enable gzip compression
  include snippets/gzip.conf;

  location / {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;

    proxy_pass http://grafana:3000/;
  }
}

Both Grafana and nginx run in Docker containers (nginx:mainline-alpine plus grafana/grafana:5.1.3).

[gaballard]

Any progress on this issue? I'm running into constant 401s when trying to access Grafana behind a reverse proxy. I can't even login - the login page returns a 401 on a POST.

[marefr]

@gaballard that sounds like a configuration error of your reverse proxy,

[marefr]

@jeroenjacobs79 are you still having this problem?

[Alex-Smart]

Is there a solution? Same stuff

server {
			listen                          80;
			server_name                     myserver.com;
			return 301                      https://$server_name$request_uri;
		}
server {
	listen                          443 ssl;
	server_name                     myserver.com;
	add_header                      Strict-Transport-Security "max-age=31536000";
	
	keepalive_timeout   			60;
	
	ssl on;
	ssl_certificate                 /etc/letsencrypt/live/myserver.com/fullchain.pem; 
	ssl_certificate_key             /etc/letsencrypt/live/myserver.com/privkey.pem;
	ssl_session_cache 				builtin:1000 shared:SSL:10m;
	ssl_session_timeout 			60m;
	
	ssl_prefer_server_ciphers 		on;
    ssl_protocols 					TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 					"HIGH:!RC4:!aNULL:!MD5:!kEDH";
	ssl_ecdh_curve 					secp384r1;
		
	location / {
		proxy_pass                              http://localhost:8080/;
		proxy_set_header Host                   $http_host;
		proxy_set_header X-Real-IP              $remote_addr;
		proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto      $scheme;
		proxy_connect_timeout      				3600;
        proxy_send_timeout						3600;
        proxy_read_timeout         				3600;
			
		satisfy                                 any;
		allow                                   127.0.0.1;
		deny                                    all;
		auth_basic                              "Username and Password Required";
		auth_basic_user_file                    /etc/nginx/.htpasswd;
	}
	location /grafana/ {
		proxy_pass                              https://localhost:3000/;
	}

[torkelo]

Closing this as it looks like a config issue in reverse proxy, not an issue in Grafana