Grafana 8 variables stopped working due to the HTTP method change

[rostrovsky]

What happened:

Background

We are trying to migrate from Grafana 7.5.13 to 8.4.1 in our k8s cluster (deployed via official Grafana helm chart).

We have a bunch of dashboards already defined for cluster monitoring and they are relying on dynamic variables obtained via Prometheus queries.

Problem

Once we swap the image to Grafana 8, dynamic variables are not working due to the lack of response from datasource proxy.

One of the cases - we need to have variable obtained via prom query:

label_values(kube_pod_info,namespace)

In grafana 7 variable queries used GET and it worked OK on our cluster:

GET https://<grafana7>/api/datasources/proxy/1/api/v1/series?match%5B%5D=kube_pod_info&start=1645620104&end=1645620404

In grafana 8 variable queries seem to be using POST:

POST https://<grafana8>/api/datasources/proxy/1/api/v1/series
form data: match%5B%5D=kube_pod_info&start=1645598928&end=1645620528

For some reason this POST hangs indefinitely without response

What you expected to happen:
POST https://<grafana8>/api/datasources/proxy/1/api/v1/series sent from Grafana UI receives response and allows configuring dynamic PromQL variables.

I would like to get some guidance how to debug this further, or how to reconfigure Grafana to use legacy GET (if possible).

1. Is it possible to configure grafana 8 to use legacy GET when querying variable values?
2. If point 1 is impossible - what can be causing this issue, and how this can be resolved?

How to reproduce it (as minimally and precisely as possible):
Not sure if this is reproducible outside of our setup. I assume if that would be common issue it would've been already posted.

Anything else we need to know?:
We use envoy proxy in every pod (including grafana & prometheus), because we use Istio service mesh on the cluster.

Initial debug from our end:

inside promehteus-server container (both curl-s work OK):

curl localhost:9090/api/v1/series?match%5B%5D=kube_pod_info&start=1645620104&end=164

curl -XPOST localhost:9090/api/v1/series --data-raw 'match%5B%5D=kube_pod_info&start=1645601212&end=1645622812'

inside grafana container (all below wget-s work OK, returning valid prometeus API payloads):

# GET full cluster FQDN
wget -O- http://prometheus-server.monitoring.svc.cluster.local/api/v1/series?match%5B%5D=kube_pod_info&start=1645620104&end=164
# GET short cluster FQDN
wget -O- http://prometheus-server.monitoring/series?match%5B%5D=kube_pod_info&start=1645620104&end=164
# GET grafana proxy endpoint
wget -O- --header='X-User-Name: <my email>' http://localhost:3000/api/datasources/proxy/1/api/v1/series?match%5B%5D=kube_pod_info&start=1645620104&end=164
# POST full cluster FQDN
wget -O- --post-data='match%5B%5D=kube_pod_info&start=1645601212&end=1645622812' http://prometheus-server.monitoring.svc.cluster.local/api/v1/series
# POST short cluster FQDN
wget -O- --post-data='match%5B%5D=kube_pod_info&start=1645601212&end=1645622812' http://prometheus-server.monitoring/api/v1/series
# POST grafana proxy endpoint
wget -O- --header='X-User-Name: <my email>' --post-data='match%5B%5D=kube_pod_info&start=1645601212&end=1645622812' http://localhost:3000/api/datasources/proxy/1/api/v1/series

Environment:

• Grafana version: 8.4.1
• Data source type & version: image bitnami/prometheus:2.31.2
• OS Grafana is installed on: official grafana/grafana8.4.1 docker image
• User OS & Browser: Windows 10, Chrome 97
• Grafana plugins: none
• Others: k8s 1.21.9

[torkelo]

can you show the raw response from

wget -O- --header='X-User-Name: <my email>' --post-data='match%5B%5D=kube_pod_info&start=1645601212&end=1645622812' http://localhost:3000/api/datasources/proxy/1/api/v1/series`

Any errors in the browser dev console?

If you are running Grafana behind a reverse proxy, you need to make sure that you are passing the original Host and Origin headers from the client request to Grafana.

All installations after Grafana v3.0-beta1 should be upgraded as soon as possible. Note that if you are running Grafana behind any reverse proxy, you need to make sure that you are passing the original Host and Origin headers from the client request to Grafana.
In the case of Apache Server, you need to add ProxyPreserveHost on in your proxy configuration. In case of NGINX, you can need to add proxy_set_header Host $http_host; in your configuration.

https://grafana.com/blog/2022/02/08/grafana-7.5.15-and-8.3.5-released-with-moderate-severity-security-fixes/

[rostrovsky]

Providing additional info:

wget from grafana container

bash-5.1$ wget -O- -S --header='X-User-Name: <email>' --post-data='match%5B%5D=kube_pod_info&start=1645601212&end=1645622812' http://localhost:3000/api/datasources/proxy/1/api/v1/series
Connecting to localhost:3000 ([::1]:3000)
  HTTP/1.1 200 OK
  Content-Security-Policy: sandbox
  Content-Type: application/json
  Date: Thu, 24 Feb 2022 17:04:04 GMT
  Server: envoy
  X-Content-Type-Options: nosniff
  X-Envoy-Upstream-Service-Time: 11
  X-Frame-Options: deny
  X-Xss-Protection: 1; mode=block
  Connection: close
  Transfer-Encoding: chunked

writing to stdout
{"status":"success","data":[...]}
100% |**********************************************************************************************************************************|  120k  0:00:00 ETA written to stdout

browser console errors:

centrifuge.js:585 WebSocket connection to 'wss://<grafana 8>/api/live/ws' failed

browser POST request copied from devtools as curl

curl 'https://<grafana 8>/api/datasources/proxy/1/api/v1/series' \
  -H 'Connection: keep-alive' \
  -H 'sec-ch-ua: " Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"' \
  -H 'accept: application/json, text/plain, */*' \
  -H 'content-type: application/x-www-form-urlencoded' \
  -H 'x-grafana-org-id: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'Origin: https://<grafana 8>' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: https://<grafana 8>/d/yd25mwB7z/new-dashboard?orgId=1&editview=templating' \
  -H 'Accept-Language: en-US,en;q=0.9' \
  -H '<cookies>' \
  --data-raw 'match%5B%5D=kube_pod_info&start=1645699856&end=1645721456' \
  --compressed

[sakjur]

Grafana 9 has a heavily rewritten Prometheus backend and Grafana 8 isn't supported for non-critical issues anymore, if this is still a problem in Grafana 9, please open a new issue for that.

[rostrovsky]

We've switched to Grafana 9.1.2 and the issue does not reproduce 😸