Unauthorized

[darox]

I'm running the latest version of grafana on two instances, but I'm facing a lot of unauthorized errors when trying to access both instances. For auth I'm currently using the built-in db, no LDAP. The data source is an influxdb.

Is this a known bug or misbehaviour?

[daniellee]

Could you give some more details:

• Are these two separate instances?
• What action triggers the unauthorized error?
• Are you getting logged out or it is just certain actions that do no work?

[torkelo]

Are they setup on different ips/domain names? if the domain name is the same and only different by port you need to have unique session cookies and remember me cookies

[darox]

-Those are seperate instances
-I don't know which action triggers the unauthorized, it just happens when I watch graphs or when accessing grafana
-Sometimes I get logged out
-Seperate domains

[pgporada]

I'm encountering this on Grafana 4.6.x with oauth through Github. It's seemingly random when I switch tabs and come back to Grafana. A refresh will "correct" the issue, but it sometimes comes back later on.

[ajardan]

I see the same issue on Grafana v4.6.2 (commit: 8db5f08), everything works as expected, and the suddenly I receive an Unauthorized warning (and some graphs are emtpy, but some show up normally).

I use Prometheus as the DataSource.

I also think this mainly happens when the dashboard is auto-refreshed, but fixes itself when I refresh it manually.

[after-the-sunrise]

Similar issue here too, but with a single Grafana instance with HTTPS, and Postgres datasource.

When the dashboard is opened, all graphs are good. But sometimes after, some of the graphs starts showing "Unauthorized" errors upon auto-refresh, but within the next (or next few) auto-refresh they recover to normal state, but then turns into "Unauthorized" state sometimes later again, repeating this random behavior on each auto-refresh.

Not sure if it's related, but found the following log messages.

lvl=eror msg="Failed to get user with id" logger=context userId=1 error="User not found"

Grafana version is as follows:

lvl=info msg="Starting Grafana" logger=server version=5.0.4 commit=7dc36ae compiled=2018-03-28T20:52:41+0900

I'm using Firefox, and I usually leave the dashboard open & untouched for multiple days, with the client machine (not the server machine hosting Grafana) going into sleep mode from time to time.

[darox]

This is not happening to me anymore with grafana 5.x

[SoulSeekkor]

I'm still having this exact same issue with Grafana 5.0.4, same messages of user not found in the log (this is with a simple local Grafana user).

[kehao95]

I'm having this issue, too. And the issue is very interesting. It may happens when I open two grafana pages of different version in the same browser and trying to do some operations.

---

I have an older version of grafana(v4.3.2 (commit: ed4d170)) and has run well on grafana.mydomain.com for a long time. Today I want to upgrade my grafana to v5.0.4. Instead of upgrade in place. I wanted to setup the new Grafana on the same machine, copy the dashboard I want , and then tear the old one down.

So what I did:

1. docker run grafana5 on the same machine of the old one with port map to 3005
2. opened the old grafana4 on grafana.mydomain.com in Safari
   And it works well
3. visit Grafana5 on grafana.mydomain.com:3005 in Safari
   So now I have two opened tab of Grafana4 and Grafana5 on my screen
4. login Grafana5, trying to do some operations .... like [create dashboard]
   Now both Grafana page crashed

Both Grafana will get Unauthorized errors and get no data points

---

Update: I changed my step 3 by visiting Grafana5 with [ip]:3005. It works fine for now.
It looks like there may be some conflicts opening two Grafana pages within the same domain.

[daniellee]

@kehao95 your use-case of in the same browser opening two Grafana instances on the same domain but with different ports is not supported. (Torkel mentioned that above).

@ajardan are your instances on the same domain or different ones?

[ajardan]

@daniellee I actually only use one instance all the time. And graphs on the dashboard I look into are pulled from 2 different datasources (Prometheus and Cloudera)

[dogada]

I also get this strange "Unauthorized" issues from time to time. Page refresh "fixes" the problem. I run Grafana v5.1.0 (844bdc5) from official Docker image. Datasource is InfluxDb. I created 2 organizations in Grafana, but use only one actually. Single 'admin' user.

[dogada]

Just got this error one more time with a new error message "Annotation query failed. Unauthorized"

[schwarzlowe]

My grafana on win10 x64 was working perfectly fine for a couple of days until I receive a warning "Unauthorized". The behavior is the same as described by @dogada and I'm also running v5.1.0 with influxdb. Both grafana and influxdb are on the same computer.

[StupidScience]

Same issue. One grafana 5.1 instance in docker. Google oauth for authorization.

Any updates?

[radium88]

Same behavior. Currently running v5.0.3 in docker, internal auth, single admin user, proxied via nginx, datasource is influxdb. Dashboard fixes itself when auto-refresh data. Mostly happens when tab long time in background

[lamoni]

Same issue seen when having two tabs open to the same instance.

[radium88]

Update to latest docker image v5.1.2 (commit: c3c690e) doesn't fixed issue

[bjacobel]

I'm having what I believe to be the same issue with Grafana 5.0.0 in Docker using GitHub OAuth. I've seen it on dashboards with InfluxDB, CloudWatch, and a mixture of both datasources. (One instance, one port, HTTPS, behind an ELB.)

Like others in this thread, I seem to see it triggered by an auto-refresh, and it goes away after a page reload. Sometimes I see the basic "Unauthorized" error message (with graph loading failures) and sometimes (more rarely) the "Annotation query failed. Unauthorized" message as well.

My suspicion is pointing towards something with the OAuth plugins? It's almost definitely due to the session backend, see below.

[bjacobel]

To add more detail I've found after digging in a little deeper, I see many errors like this in my logs:

t=2018-05-16T16:55:39+0000 lvl=eror msg="Failed to get user with id" logger=context userId=2 error="User not found"

The only place I see such an error thrown is in this line of code, which seems related to managing sessions and session cookies?

grafana/pkg/middleware/middleware.go

Line 97 in 0ad6336

ctx.Logger.Error("Failed to get user with id", "userId", userId, "error", err)

I'm storing my sessions using the default file backend, but via a mounted EFS share, I wonder if that is a potential complication.

[harshitha-m]

I Faced this issue when i try to open two different Grafana (which are Running in Different port )in the same browser.
I get Unauthorized Errors and Sometimes get logged out

[marefr]

It would be really interesting to see what SQL queries are executed when you receive the Failed to get user with id log message. If you easily can reproduce this it would be super valuable if you could enable logging of sql queries and report back your findings:

[database]
# Set to true to log the sql calls and execution times.
log_queries = true

Thank you

[bjacobel]

@marefr It seems like these errors always occur surrounded by one of these two queries:

SELECT\n\t\tu.id as user_id,\n\t\tu.is_admin as is_grafana_admin,\n\t\tu.email as email,\n\t\tu.login as login,\n\t\tu.name as name,\n\t\tu.help_flags1 as help_flags1,\n\t\tu.last_seen_at as last_seen_at,\n\t\t(SELECT COUNT(*) FROM org_user where org_user.user_id = u.id) as org_count,\n\t\torg.name as org_name,\n\t\torg_user.role as org_role,\n\t\torg.id as org_id\n\t\tFROM `user` as u\n\t\tLEFT OUTER JOIN org_user on org_user.org_id = 1 and org_user.user_id = u.id\n\t\tLEFT OUTER JOIN org on org.id = org_user.org_id WHERE u.id=? []interface

UPDATE `user` SET `last_seen_at` = ? WHERE `id`=? []interface

Full example logs:

t=2018-05-30T15:59:39+0000 lvl=info msg="[SQL] SELECT\n\t\tu.id as user_id,\n\t\tu.is_admin as is_grafana_admin,\n\t\tu.email as email,\n\t\tu.login as login,\n\t\tu.name as name,\n\t\tu.help_flags1 as help_flags1,\n\t\tu.last_seen_at as last_seen_at,\n\t\t(SELECT COUNT(*) FROM org_user where org_user.user_id = u.id) as org_count,\n\t\torg.name as org_name,\n\t\torg_user.role as org_role,\n\t\torg.id as org_id\n\t\tFROM `user` as u\n\t\tLEFT OUTER JOIN org_user on org_user.org_id = 1 and org_user.user_id = u.id\n\t\tLEFT OUTER JOIN org on org.id = org_user.org_id WHERE u.id=? []interface
{}
{2} - took: 54.517418ms" logger=sqlstore.xorm
t=2018-05-30T15:59:39+0000 lvl=info msg="[SQL] UPDATE `user` SET `last_seen_at` = ? WHERE `id`=? []interface
{}
{\"2018-05-30 15:59:39\", 2} - took: 42.957209ms" logger=sqlstore.xorm
t=2018-05-30T15:59:39+0000 lvl=eror msg="Failed to get user with id" logger=context userId=2 error="User not found"
t=2018-05-30T15:59:39+0000 lvl=info msg="[SQL] SELECT\n\t\tu.id as user_id,\n\t\tu.is_admin as is_grafana_admin,\n\t\tu.email as email,\n\t\tu.login as login,\n\t\tu.name as name,\n\t\tu.help_flags1 as help_flags1,\n\t\tu.last_seen_at as last_seen_at,\n\t\t(SELECT COUNT(*) FROM org_user where org_user.user_id = u.id) as org_count,\n\t\torg.name as org_name,\n\t\torg_user.role as org_role,\n\t\torg.id as org_id\n\t\tFROM `user` as u\n\t\tLEFT OUTER JOIN org_user on org_user.org_id = 1 and org_user.user_id = u.id\n\t\tLEFT OUTER JOIN org on org.id = org_user.org_id WHERE u.id=? []interface
{}
{2} - took: 69.013955ms" logger=sqlstore.xorm
t=2018-05-30T15:59:39+0000 lvl=info msg="[SQL] UPDATE `user` SET `last_seen_at` = ? WHERE `id`=? []interface
{}
{\"2018-05-30 15:59:39\", 2} - took: 5.593997ms" logger=sqlstore.xorm
t=2018-05-30T15:59:39+0000 lvl=eror msg="Failed to get user with id" logger=context userId=2 error="User not found"
t=2018-05-30T15:59:39+0000 lvl=eror msg="Failed to get user with id" logger=context userId=2 error="User not found"
t=2018-05-30T15:59:39+0000 lvl=info msg="[SQL] UPDATE `user` SET `last_seen_at` = ? WHERE `id`=? []interface
{}
{\"2018-05-30 15:59:39\", 2} - took: 46.673µs" logger=sqlstore.xorm
t=2018-05-30T15:59:39+0000 lvl=eror msg="Failed to get user with id" logger=context userId=2 error="User not found"
t=2018-05-30T15:59:39+0000 lvl=eror msg="Failed to get user with id" logger=context userId=2 error="User not found"
t=2018-05-30T15:59:39+0000 lvl=info msg="[SQL] UPDATE `user` SET `last_seen_at` = ? WHERE `id`=? []interface
{}
{\"2018-05-30 15:59:39\", 2} - took: 621.538µs" logger=sqlstore.xorm

[marefr]

Thanks a lot @bjacobel. Everything looks good here according to me. There's an actual user id provided all the way down to the database query. Really strange. Starting to think there's a bug with our 3rd party database lib xorm.

Did you do anything specific to generate those log messages?
What database are you using? What session storage?
What request is resulting in unauthorized, you can enable router logging to log all requests:

[server]
router_logging = true

[benkeil]

We have the same error on 5.1.4 in Kubernetes.

[bjacobel]

Hi @marefr, sorry, I forgot to respond with the additional requested detail.

Did you do anything specific to generate those log messages?

The queries are generated by loading a dashboard and then waiting for an auto-refresh. It doesn't happen on every auto-refresh, and sometimes it can trigger with a manual click of the dashboard refresh button (the one built into Grafana, not the browser refresh button) but generally it seems to happen more often when the user is inactive (leaving grafana in a background tab, for example.)

What database are you using? What session storage?

The database is SQLite on a mounted NFS (EFS) share, and the session storage is the default (file), although I have also tried the memory-based storage and it also had the same issue. We have one grafana host behind a load balancer, and I've enabled session stickiness on that load balancer.

What request is resulting in unauthorized?

I didn't enable router logging, because I can see the request that is resulting in unauthorized from the browser:

[Some sensitive information redacted]

Request URL: https://[my grafana hostname]/api/tsdb/query
Request Method: POST
Status Code: 401 
Remote Address: [my load balancer IP]:443
Referrer Policy: no-referrer-when-downgrade
:authority: [my grafana hostname]
:method: POST
:path: /api/tsdb/query
:scheme: https
accept: application/json, text/plain, */*
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: no-cache
content-length: 478
content-type: application/json;charset=UTF-8
cookie: _ga=GA1.2.1782868908.1520436196; __gads=ID=b1c7d78e4fd8b9fb:T=1520436200:S=ALNI_MYT2aRMJqYtHY-CkgaPWmuNtsGEtA; sailthru_hid=919b24e8c99698a8b1829b81eda7135a5956a753dd4c29265f8b45b3a11fb749fc11562ad2abbb1220b9ef37; grafana_sess=[16-char hexadecimal session string]; AWSALB=IUyH6LlTXI/TJlteL8pr838fC7nsvth7s63o5WzqOa6wsCPRpHg20vYurCrYpbIWci27fQtzQpoRxVlIc8Ud/rEPIJvqWvT21an4e9aQmZioTEAFHA3+iWv7bPHs
dnt: 1
origin: https://[my grafana hostname]
pragma: no-cache
referer: https://[my grafana hostname]/d/[dashboard path]?refresh=5m&orgId=1&from=now-1h&to=now
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36
x-grafana-org-id: 1

[davewat]

Hi @marefr, sorry, I forgot to respond with the additional requested detail. ...

@bjacobel this is likely unrelated to the specific issue, however SQLite’s developers recommend not running SQLite over NFS. Specifically, the Grafana process should not access the DB over an NFS mount, and running from any networked file system without strong file lock support is not recommended.

On a side note, we use SQLite with session storage as you do, but on local file system. We have not experienced this same issue.

We have also tweaked the SQLite config in grafana to use WAL mode (of which I’ll eventually do a PR) for better performance.

_{Sent with GitHawk}

[isclever]

I'm having the same issue in my docker Grafana and InfluxDB stack.
Grafana v5.1.3 (commit: 0871432)
InfluxDB 1.5.3

Grafana is using local storage via docker volumes with sqlite database. Volumes are using local SSD.
I get the error every time I leave the tab for more then a few minutes. If I leave dev tools up in Firefox I see:

GET http://x.x.x.x:3000/api/datasources/proxy/1/query?db=(Redacted info)
{"message":"Unauthorized"}

Any sort of refresh clears the errors.

[ggggh]

Solved for me after a Raspbian upgrade which took me to Postgres 9.6 (from 9.4). Grafana still on 5.4.3

Forget what I said...it's back. Less often, I'd say...but still happening.

[devanshkv]

@ggggh any solutions? It just started happening out of the blue for me!

[ggggh]

@ggggh any solutions? It just started happening out of the blue for me!

Nothing...! It cleared out with the postgres version upgrade, and seems to be coming back again, more often each day

[devanshkv]

@ggggh Thanks!
I've switched to Postgres, but that isn't helping either :(

[botzill]

having the same issues using Grafana 6.2.1 and Postgress 11, but this is happening only on dashbaords I load from JSON and then try to access them.

Any updates on this?

[botzill]

OK, I found the issue in my case. My PG had a limited number of connections and in grafana max_open_conn was not set. After I did set this option, it works OK.

[syardumian-chc]

Same is happening for me on Grafana 6.1.6 and packaged-in SQLite DB. This problem breaks our internal dev efforts for customizing Grafana. Changing max_open_conn does not work (though I didn't expect it to since it was a fix for Postgres).

[qhartman]

The root cause of this seems to be grafana trying to connect to the underlying DB when authenticating, but failing to do so. With SQLite, that will happen often and at a low count of concurrent usage since SQLite locks so aggressively. In most cases, migrating to a real RDBMS (I like postgres) will solve the issue. It's possible it can come up again if you run into a connection limit (or similar) problem, but that's a DB concern more than a Grafana concern. If you're using Grafana for anything other than a demo, you should back it with a real DB. If that DB is configured correctly for your usage, that should solve this problem.

…

On Mon, Jun 10, 2019 at 11:20 AM syardumian-chc ***@***.***> wrote: Same is happening for me on Grafana 6.1.6 and packaged-in SQLite DB. This problem breaks our internal dev efforts for customizing Grafana. Changing max_open_conn does not work (though I didn't expect it to since it was a fix for Postgres). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10727?email_source=notifications&email_token=AAAK6YSUDLXPF2E4436CEOTPZ2EMFA5CNFSM4EO23EH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXKQ3UY#issuecomment-500501971>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAK6YQLR3FSCNEQR7SNEKLPZ2EMFANCNFSM4EO23EHQ> .

[ggggh]

I've increased the connection limit and the max idle connections, but still keep hitting this issue randomly. Not just that, but dashboards which have been open for a while seem to get slower and slower to refresh, with the loading-gif evident on each panel and slowly disappearing sequentially as each panel completes loading. It's fine if I close the browser window and open a new one. I guess my dashboard has got more complex, but that doesn't explain why a fresh load of the page "fixes it".

[naturalbeau]

I am getting random error too. Really do not know what is the issue. Using IP address seems fine, but with the kubeneters ingress, it shows the "annotation query failed" randomly.

[kmott]

FWIW, I recently switched my ingress loadbalancer to Fabio (from Traefik) and updated Grafana (Docker image, no additional database backends) to v6.4.2, and the 401 unauthorized errors seem to have gone away when doing automatic refresh (interval set to 10 seconds, running all day). It's unlikely that switching to Fabio fixed the issue, I'm guessing it was the newer version of Grafana that helped, but I'm not 100% sure.

[torkelo]

Closing this as there are no new reports recently. if you think there still is an issue please open a new issue

[ikkerens]

I recently installed grafana on my kubernetes cluster and ran into a similar issue.
I'm using docker image grafana/grafana:6.4.3

Checking my pod logs, I found this interesting little tidbit:

t=2019-11-01T15:18:33+0000 lvl=info msg="Successful Login" logger=http.server User=--snip--
t=2019-11-01T15:19:09+0000 lvl=eror msg="Failed to look up user based on cookie" logger=context error="dial tcp: lookup postgres.databases.svc.cluster.local: no such host"
t=2019-11-01T15:19:09+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/datasources/proxy/1/query status=401 remote_addr=--snip-- time_ms=11 size=26 referer="https://--snip--/d/TuobtjoZz/--snip--?orgId=1&refresh=5s&from=now-12h&to=now"

DNS issues are not something I've encountered before within my cluster, but I did some googling and found this particular issue: kubernetes/kubernetes#30215

Would it be possible for grafana to ship both alpine and non-alpine images like a lot of docker images do? Seems like that would resolve the issue.
If there's anything I can do in testing this or helping debug let me know, I'll provide information as requested.

[ikkerens]

After downgrading to 6.3.6 (which isn't alpine-based) the issue disappeared entirely on my end.

[n0-bs]

I faced the same issue , two separate Grafana (containers) open in the same browser
when login to the second the first ask me to login again , login to the first the second ask me to login again
can't keep both login
the solution I found is to change in one of the Grafana default.ini file
login_cookie_name = grafana_session
to
login_cookie_name = grafana_session_1
restart the container and the browser , now it working fine
for now I keep the file out side the container
need to set this parameter when creating the container

[marefr]

@ikkerens please try the ubuntu based image then, 6.6.2-ununtu

[marefr]

@n0-bs I'm sorry but if you're running multiple instances of Grafana it's suggested to use MySQL or Postgres as database.

[n0-bs]

Sorry, but how , use of MySQL or Postgres as database., will solve the cookie conflict when I open these two different Grafana instances in the same browser , I'm not talking about HA case
I have two different Grafana instances (containers) on the same server

[helderco]

I'm still seeing this with 6.7.2. I upgraded from 6.5 to 6.6, then 6.7. Using docker with PostgreSQL, tried 6.7.2 image then 6.7.2-ubuntu.

This is the error I'm getting in the logs:
lvl=eror msg="Failed to look up user based on cookie" logger=context error="pq: remaining connection slots are reserved for non-replication superuser connections"

[helderco]

Fixed (at least for now) by restarting postgres.

[emzfuu]

Im using the latest version of Grafana and still seeing the unauthorized issue eveytime I access it. Im using Grafana in kubernetes. I deployed it in 3 different pod in 3 different nodes. Im using the native database of it. Any suggestion to fix the isssue?

[bergquist]

@emzfuu If you run multiple instances you need to point all of them to the same database. mysql/postgres

[emzfuu]

@bergquist is there any other way to fix the issue?

[emzfuu]

Just to elaborate my question above im using 3 different Grafana (stand alone) which is being access through single load balancer. The 3 Grafana has their own db (sqlite3). Every time I access it I receive the unauthorize error.

[linux0x5c]

I have same problem,use nfs.

[clever-trevor]

Not sure if this is the same issues are above but in my case

• Start Chrome
• In one tab, connect to Grafana instance A (192.168.1.100:3000)
• In separate tab, connect to Grafana instance B (192.168.1.100:3001)
• Log into instance A. Works
• Log into instance B. Works.
• Go back to instance A and try any feature in the browser, and you are redirected back to login screen.
• Re-login to instance A. Works
• Go to instance B and you find out you are logged out from there.

This only seems to be a problem if both instances are on the same IP Address.

[marefr]

@schmorgs you have to change the login_cookie_name in of the instances for this to work. Reason is that the login cookie would get the same domain (IP), name and path so they would interfere.

[clever-trevor]

@schmorgs you have to change the login_cookie_name in of the instances for this to work. Reason is that the login cookie would get the same domain (IP), name and path so they would interfere.

Nice one!!!