unauthorized 401 for endpoint /api/datasources/name/foo?orgId=bar

[limscoder]

What Grafana version are you using?

5.1.0 - Docker image

What datasource are you using?

Prom

What OS are you running grafana on?

Running multiple pods in GKE Kubernetes on ContainerOptimizedOS nodes

What did you do?

GET request to endpoint from script (not through UI): /api/datasources/name/foo?orgId=bar
I'm using basic auth as super admin user and admin user's orgId does NOT match orgId in query params

What was the expected result?

200 response

What happened instead?

Sometimes it works, sometimes it returns a 401. It appears to randomly return 200 or 401 when the same request is repeated in a loop.

Maybe related

#11757
#11715

[marefr]

• Would it be possible for you to provide Grafana server log with entries around the time when you receive a 401?
• The request you're doing is a HTTP GET?
• What database are you using for configuration and session, sqlite, mysql, postgres or something else?

[limscoder]

GET request
Config is in Postgres on Google CloudSQL

Here is a log entry for a failed request:

grafana-bd8d7575f-6w4mw grafana t=2018-05-04T15:47:40+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/datasources/name/{foo} status=401 remote_addr="{ip}, {ip}" time_ms=9 size=26 referer="http://127.0.0.1:9096/api/datasources/name/{foo}?orgId={bar}"

[limscoder]

I also get some 302 responses in the logs, but I don't see these in the client, so perhaps they are being handle by kube-proxy.

grafana-bd8d7575f-5vgdv grafana t=2018-05-04T16:58:49+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=2 uname=admin method=GET path=/api/datasources/name/{foo} status=302 remote_addr=127.0.0.1 time_ms=60 size=99 referer=

[limscoder]

login_attempt and user_auth DB tables are both empty after the failures.

[marefr]

@limscoder thanks.

admin user's orgId does NOT match orgId in query params

Why doesn't it match? Have you tried making them match or remove the orgId querystring param alltogether?

How often are you running these scripts? Are the scripts calling specific pods, load-balanced endpoint and/or from outside kubernetes?

The 302 status tells me that Grafana is responding with a redirect. This may be due to how you configured Grafana. Can you please include your server section from the Grafana configuration ?

Would it be possible for you to change the log level to debug, see documentation. And then see if the logs contain any other interesting information.

[limscoder]

Why doesn't it match? Have you tried making them match or remove the orgId querystring param alltogether?

I think I was incorrectly understanding how users work in Grafana. This user has admin permissions in all orgs.

How often are you running these scripts? Are the scripts calling specific pods, load-balanced endpoint and/or from outside kubernetes?

This is a command line tool that loops through all of our Grafana orgs and hits some API endpoints to configure them. It's currently calling specific pods from outside of kubernetes via kubectl port-forwarding, although we are currently working on running it from within kubernetes and hitting the load-balanced service endpoint, so I can hopefully update on whether that's any different soon.

Here's the server config:

    [server]
    ;protocol = http
    ;http_addr =
    ;http_port = 3000
    ;domain = localhost
    ;enforce_domain = false
    root_url = https://{reverse-proxy-domain}/
    ;router_logging = false
    ;static_root_path = public
    ;enable_gzip = false
    ;cert_file =
    ;cert_key =

I'll try to get some debug logs.

[limscoder]

Belated update:

I see no difference in behavior or logs when running within the Kube cluster vs running outside with Kubectl, and I also don't see any additional log information with debug level. I see the 302 and 401 requests listed in a comment above in the logs and nothing else. Behavior seems to be specific to the datasource endpoint.

[marefr]

Feels a bit similar to #10727 - do you see any Failed to get user with id log messages related to your unauthorized log message?

[daniellee]

Have you tried switching the org of the admin user:

curl -X POST http://admin:admin@localhost:3000/api/user/using/<id of the org>

http://docs.grafana.org/http_api/user/#switch-user-context-for-signed-in-user

[torkelo]

orgId is not a valid query parameter to this api call so closing this