Popular graphana dashboards show no data for "Pods Memory usage" and "Pods CPU usage"

[surfingjoe]

Running bare-metal K8s with flannel network and MetalLB as Load Balancer.
Cloned https://github.com/prometheus-operator/kube-prometheus.git just a few days ago. Created "monitoring" namespace. Performed Kubectl apply for manifests/setup, waited till all pods are up, then applied manifests/

Instead of port forwarding, I patched each service to be: "Type" load-balancer considering that I'm using MetalLB.

Everything is operational. Can successfully deploy containers with services exposed externally by the load balancer MetalLB. All is well in the world. Can open Prometheus port 9090 & Graphana port 3000 externally from a browser and everything seems operational. Graphana is seeing Prometheus data, and community dashboards are seeing some of the Prometheus data.

Import popular community dashboards like Community dashboard number 315 for example. Oops, the dashboard reports "No Data" for PODS memory and CPU usage, but Graphana can see Cluster Memory & CPU usage. The dashboard can see Network I/O pressure, but "No data" for Pods network I/O.

As I'm a newbie to Graphana, I'm hoping someone wiser than me can help :)

[bryanasdev000]

Running bare-metal K8s with flannel network and MetalLB as Load Balancer. Cloned https://github.com/prometheus-operator/kube-prometheus.git just a few days ago. Created "monitoring" namespace. Performed Kubectl apply for manifests/setup, waited till all pods are up, then applied manifests/

Instead of port forwarding, I patched each service to be: "Type" load-balancer considering that I'm using MetalLB.

Everything is operational. Can successfully deploy containers with services exposed externally by the load balancer MetalLB. All is well in the world. Can open Prometheus port 9090 & Graphana port 3000 externally from a browser and everything seems operational. Graphana is seeing Prometheus data, and community dashboards are seeing some of the Prometheus data.

Import popular community dashboards like Community dashboard number 315 for example. Oops, the dashboard reports "No Data" for PODS memory and CPU usage, but Graphana can see Cluster Memory & CPU usage. The dashboard can see Network I/O pressure, but "No data" for Pods network I/O.

As I'm a newbie to Graphana, I'm hoping someone wiser than me can help :)

In such cases, you can check the following items:

• If all Prometheus targets are OK (In Prometheus UI, click Status -> Targets);
• If the dashboard is not too "old" and therefore uses the old name of some metric (3rd party dashboards from Grafana.com);
• If you can manually run the query from this panel on Prometheus (Open the panel, get the metric name and try to run it manually in the Prometheus UI);

In the case of Pod-specific CPU metrics, I would particularly look at the kube-state-metrics target, both the target in Prometheus and Pod logs.

[surfingjoe]

An example is to import Grafana dashboard number 315. Some of the graphs work, and some show "no data." One of the metrics is Pods CPU Usage sum (rate (container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^.*$"}[1m])) by (pod_name)
I found the reason for "No Data" in that panel. It is expecting "k8s" in the name. I edited and removed k8s_ where it says name="k8s_.*" and that one measurement started working instead of saying "no data." That is one fix. There are other "no data" panels I have yet to fix.
My problem is that I do not know enough about the query language to understand what works or does not work. I would love to see a good write-up by someone who explains and provides query language examples for monitoring Kubernetes. I can't seem to find a good book or open a web page that offers many query language examples. However, out of the box Grafana dashboard number 315 is pretty good.

This dashboard is too old, isn't updated since 2017 (https://grafana.com/grafana/dashboards/315/revisions), this may be one of your problems.

Did you check the default provisioned dashboards (https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/dashboards)?

Let me start by admitting that after looking at Grafana for days now, I've only just discovered the "built-in" dashboards for Kubernetes monitoring because you pointed out they exist. I did not know the default dashboards existed because, by default, browsing dashboards show the most recently viewed dashboards. Every day until today, when I browse the list of dashboards, the only dashboards I see are the dashboards imported from Grafana.com. THAT IS THE PRIMARY REASON FOR MY ISSUE.

When browsing the dashboards, you have to click on the hamburger icon next to the folder icon to see all possible dashboards. Ugh, I have only used the "default" list until today. Because I assumed the hamburger icon would be a "list view" and not a view that changes what is in the list.

So thank you for pointing out that built-in dashboards exist. Saying they exist challenged me to look in every nook and cranny of Grafana to find the built-in dashboards.

[bespokeeagle]

Regarding your question "I want to confirm I am looking at the right thing". I assume you are using Grafana's import utility to import a dashboard from Grafana.com. Specifically dashboard number 315, (which bryanasdev000 pointed out that it hasn't been updated since 2017). The fourth row in that dashboard is Pods CPU Usage. You can see the query used by that panel, by clicking the down arrow to the right of the title and then either edit or explore. Both options are "edit or explore", let me see the query. I hope that helps you validate that we are looking at the same thing, which is the query for the fourth row of an imported dashboard from Grafana.com, number 315.

actually i did not import any dashboards from grafana.

i am using the default dashboards here https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/dashboards

here's what my query looks like sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster="$cluster", node=~"$node"}) by (pod)

[bryanasdev000]

Regarding your question "I want to confirm I am looking at the right thing". I assume you are using Grafana's import utility to import a dashboard from Grafana.com. Specifically dashboard number 315, (which bryanasdev000 pointed out that it hasn't been updated since 2017). The fourth row in that dashboard is Pods CPU Usage. You can see the query used by that panel, by clicking the down arrow to the right of the title and then either edit or explore. Both options are "edit or explore", let me see the query. I hope that helps you validate that we are looking at the same thing, which is the query for the fourth row of an imported dashboard from Grafana.com, number 315.

actually i did not import any dashboards from grafana.

i am using the default dashboards here https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/dashboards

here's what my query looks like sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster="$cluster", node=~"$node"}) by (pod)

In this case, you can make the checks that I pointed in #1694 (comment)

Also, you are checking which dashboard? Kubernetes / Compute Resources / Pod ?

Which version of kube-prometheus are you running?

EDIT: Remember to remove Grafana placeholders when running the query in Prometheus like $cluster.

[bespokeeagle]

Regarding your question "I want to confirm I am looking at the right thing". I assume you are using Grafana's import utility to import a dashboard from Grafana.com. Specifically dashboard number 315, (which bryanasdev000 pointed out that it hasn't been updated since 2017). The fourth row in that dashboard is Pods CPU Usage. You can see the query used by that panel, by clicking the down arrow to the right of the title and then either edit or explore. Both options are "edit or explore", let me see the query. I hope that helps you validate that we are looking at the same thing, which is the query for the fourth row of an imported dashboard from Grafana.com, number 315.

actually i did not import any dashboards from grafana.
i am using the default dashboards here https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/dashboards
here's what my query looks like sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster="$cluster", node=~"$node"}) by (pod)

In this case, you can make the checks that I pointed in #1694 (comment)

Also, you are checking which dashboard? Kubernetes / Compute Resources / Pod ?

Which version of kube-prometheus are you running?

EDIT: Remember to remove Grafana placeholders when running the query in Prometheus like $cluster.

Thanks for the checks, i actually do not have access to a prometheus UI to run some of the checks
Also the dashboard i am checking is Kubernetes / Compute Resources / Node (Pods)
my kube-prometheus version is 0.9
on removing the grafana placeholders do you mean i should take it out of the query like
sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster, node}) by (pod)

[bryanasdev000]

Regarding your question "I want to confirm I am looking at the right thing". I assume you are using Grafana's import utility to import a dashboard from Grafana.com. Specifically dashboard number 315, (which bryanasdev000 pointed out that it hasn't been updated since 2017). The fourth row in that dashboard is Pods CPU Usage. You can see the query used by that panel, by clicking the down arrow to the right of the title and then either edit or explore. Both options are "edit or explore", let me see the query. I hope that helps you validate that we are looking at the same thing, which is the query for the fourth row of an imported dashboard from Grafana.com, number 315.

actually i did not import any dashboards from grafana.
i am using the default dashboards here https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/dashboards
here's what my query looks like sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster="$cluster", node=~"$node"}) by (pod)

In this case, you can make the checks that I pointed in #1694 (comment)
Also, you are checking which dashboard? Kubernetes / Compute Resources / Pod ?
Which version of kube-prometheus are you running?
EDIT: Remember to remove Grafana placeholders when running the query in Prometheus like $cluster.

Thanks for the checks, i actually do not have access to a prometheus UI to run some of the checks Also the dashboard i am checking is Kubernetes / Compute Resources / Node (Pods) my kube-prometheus version is 0.9 on removing the grafana placeholders do you mean i should take it out of the query like sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster, node}) by (pod)

You can use Grafana Explore on your Prometheus Datasource to test a few queries, but without actual access to Prometheus UI will be hard to debug if the Prometheus itself is getting metrics from Kubernetes.

https://grafana.com/docs/grafana/latest/explore/

EDIT: About the query, you just remove the filter, so it will be something like this:

sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{}) by (pod)