Grafana not showing any data from the cluster even though it is shown on k8 dashboard

[kskalski]

Issue

I'm struggling to get Grafana dashboard to show any data.

Environment

I have kubernetes cluster created on my own VM with kubeadm 1.5 and I installed heapster components (not created by default with kubeadm) based on config from https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb (I had to fix the image names as per kubernetes-retired/heapster#1478)
So right now I have

Dashboard version: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.0 
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:52:01Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Operating system: Debian Jessie
Heapster version: gcr.io/google_containers/heapster:v1.3.0-beta.0
Grafana version: gcr.io/google_containers/heapster-grafana-amd64:v4.0.2
Influxdb version: gcr.io/google_containers/heapster-influxdb-amd64:v1.1.1

Steps to reproduce

Entering grafana homepage, clicking through Cluster (/dashboard/db/cluster) or Pods (/dashboard/db/pods) dashboard, changing filtering options, etc.

Observed result

Whatever I do I get "No datapoints"
I verified that the requests are correctly issued to influxdb (its logs contain e.g.
2017-01-23T14:44:22.674597457Z [httpd] 10.32.3.157 - root [23/Jan/2017:14:44:22 +0000] "GET /query?db=k8s&epoch=ms&q=SELECT+sum%28%22value%22%29+FROM+%22memory%2Fusage%22+WHERE+%22type%22+%3D+%27node%27+AND+time+%3E+now%28%29+-+30m+GROUP+BY+time%282s%29+fill%28null%29%3BSELECT+sum%28%22value%22%29+FROM+%22memory%2Fworking_set%22+WHERE+%22type%22+%3D+%27node%27+AND+time+%3E+now%28%29+-+30m+GROUP+BY+time%282s%29+fill%28null%29%3BSELECT+sum%28%22value%22%29+FROM+%22memory%2Flimit%22+WHERE+%22type%22+%3D+%27node%27+AND+time+%3E+now%28%29+-+30m+GROUP+BY+time%282s%29+fill%28null%29%3BSELECT+sum%28%22value%22%29+FROM+%22memory%2Frequest%22+WHERE+%22type%22+%3D+%27node%27+AND+time+%3E+now%28%29+-+30m+GROUP+BY+time%282s%29+fill%28null%29 HTTP/1.1" 200 49 "http://bigdata2:30081/dashboard/db/cluster"

I can also see that heapster can connect to influxdb, since there are log entries like:
2017-01-23T14:58:05.057698055Z [httpd] 10.32.3.153 - root [23/Jan/2017:14:58:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 500 72 "-" "heapster/v1.3.0-beta.0" 58082bd2-e17c-11e6-8262-000000000000 6618 2017-01-23T14:58:05.057788774Z [httpd] 10.32.3.153 - root [23/Jan/2017:14:58:05 +0000] "GET /ping HTTP/1.1" 204 0 "-" "heapster/v1.3.0-beta.0" 58094b31-e17c-11e6-8263-000000000000 14

Moreover, Kubernetes dashboard itself shows correct data from Heapster (each pod has CPU / Memory usage level, at the top there is also graph with resource usage for all pods, etc.).

Expected result

Grafana dashboard should be able to display data like cpu, memory, etc. historical usage.

Comments

At this point I don't see any errors in either Heapster, Grafana or Influxdb components, they query each other fine, there is simply no data even though Heapster seems to be also correctly collecting data from apiserver (gets displayed on dashboard). I also checked manually issuing query to influxdb and it indeed returns empty set {} for all queries.

[webner]

I run into the same problem. The workaround was to create a default retention policy in influxdb as suggested here kubernetes-retired/heapster#1348 (comment)

[TomasBahnik]

The same problem with missing data in grafana. Can you advise me, pls, how can I set the retention policy when there is no influx CLI command nor curl in /bin directory of gcr.io/google_containers/heapster-influxdb-amd64:v1.1.1 image?

[TomasBahnik]

Solved by

curl -i -XPOST 'http://{influxdb_cluster_ip}:8086/query' --data-urlencode 'q=CREATE RETENTION POLICY "default" ON "k8s" DURATION INF REPLICATION 1 DEFAULT'
HTTP/1.1 200 OK
Connection: close
Content-Type: application/json
Request-Id: c19c946d-e3bb-11e6-8c84-000000000000
X-Influxdb-Version: unknown
Date: Thu, 26 Jan 2017 11:37:02 GMT
Transfer-Encoding: chunked

{"results":[{}]}

from heapster pod

kubectl exec -ti -n kube-system heapster-3821066434-pun2p -- sh

[TomasBahnik]

Note that heapster image has DNS issue (see /issues/30215) and internal influxdb IP must be used.

[kskalski]

Confirmed this solves my problem

[xmik]

Thanks @Slouly, I confirm that setting default retention policy solves this. Instead of INF, I set it to 30d and it worked too. (Verification: measurements like cpu/limit are visible on influxdb).

[ChrisBuchholz]

I can confirm this also fixed the problem for me, @Slouly.

[roberthbailey]

It sounds like this issue has been resolved, so I'm going to close it.

[concaf]

from heapster pod
kubectl exec -ti -n kube-system heapster-3821066434-pun2p -- sh

FWIW, I hit this issue today and I'm unable to exec into the heapster pod since they have now based it off scratch and it has no sh, bash, curl, etc binaries.

[brendandburns]

@containscafeine you should be able to run that 'curl' command from any container in the cluster, e.g.

kubectl run -it --rm busybox --image busybox