Grafana not showing any data from the cluster even though it is show in dashboard

[kskalski]

Issue

I'm struggling to get the grafana dashboard to show any data.

Environment

I have kubernetes cluster created on my own VM with kubeadm 1.5 and I installed heapster components (not created by default with kubeadm) based on config from https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb (I had to fix the image names as per kubernetes-retired/heapster#1478)
So right now I have

Dashboard version: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.0 
Kubernetes version: 1.5
Operating system: Debian Jessie
Heapster version: gcr.io/google_containers/heapster:v1.3.0-beta.0
Grafana version: gcr.io/google_containers/heapster-grafana-amd64:v4.0.2
Influxdb version: gcr.io/google_containers/heapster-influxdb-amd64:v1.1.1

Steps to reproduce

Entering grafana homepage, clicking through Cluster or Pods dashboard, changing filtering options, etc.

Observed result

Whatever I do I get "No datapoints"
I verified that the requests are correctly issued to influxdb (its logs contain e.g.
2017-01-23T14:44:22.674597457Z [httpd] 10.32.3.157 - root [23/Jan/2017:14:44:22 +0000] "GET /query?db=k8s&epoch=ms&q=SELECT+sum%28%22value%22%29+FROM+%22memory%2Fusage%22+WHERE+%22type%22+%3D+%27node%27+AND+time+%3E+now%28%29+-+30m+GROUP+BY+time%282s%29+fill%28null%29%3BSELECT+sum%28%22value%22%29+FROM+%22memory%2Fworking_set%22+WHERE+%22type%22+%3D+%27node%27+AND+time+%3E+now%28%29+-+30m+GROUP+BY+time%282s%29+fill%28null%29%3BSELECT+sum%28%22value%22%29+FROM+%22memory%2Flimit%22+WHERE+%22type%22+%3D+%27node%27+AND+time+%3E+now%28%29+-+30m+GROUP+BY+time%282s%29+fill%28null%29%3BSELECT+sum%28%22value%22%29+FROM+%22memory%2Frequest%22+WHERE+%22type%22+%3D+%27node%27+AND+time+%3E+now%28%29+-+30m+GROUP+BY+time%282s%29+fill%28null%29 HTTP/1.1" 200 49 "http://bigdata2:30081/dashboard/db/cluster"

I can also see that heapster can connect to influxdb, since there are log entries like:
2017-01-23T14:58:05.057698055Z [httpd] 10.32.3.153 - root [23/Jan/2017:14:58:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 500 72 "-" "heapster/v1.3.0-beta.0" 58082bd2-e17c-11e6-8262-000000000000 6618 2017-01-23T14:58:05.057788774Z [httpd] 10.32.3.153 - root [23/Jan/2017:14:58:05 +0000] "GET /ping HTTP/1.1" 204 0 "-" "heapster/v1.3.0-beta.0" 58094b31-e17c-11e6-8263-000000000000 14

Moreover, Kubernetes dashboard itself shows correct data from Heapster (each pod has CPU / Memory usage level, at the top there is also graph with resource usage for all pods, etc.).

Expected result

Grafana dashboard should be able to display data like cpu, memory, etc. historical usage.

Comments

At this point I don't see any errors in either of Heapster, Grafana, Influxdb components, they query each other fine, there is simply no data even though Heapster seems to be also correctly collecting data from apiserver (gets displayed on dashboard). I also checked manually issing query to influxdb and it indeed returns empty set {} for all queries.

[maciaszczykm]

Hi! Issue seems to be related to Kubernetes and its configuration, not the Dashboard:

Kubernetes dashboard itself shows correct data from Heapster (each pod has CPU / Memory usage level, at the top there is also graph with resource usage for all pods, etc.).

Could you report it here?

[kskalski]

Ok, I guess it is a problem between Heapster and InfluxDb data propagation, but if you say generic k8 tracker is better for this here it is: kubernetes/kubernetes#40346