Some of node's metrics are missing on Nodes overview tab

[amravyan]

Hi!

Facing a strange issue: some of node's metrics (pod limits usage percent, for example) are missing on the tab Nodes overview (N/A NaN%). On the other hand, all these metrics are available in prometheus.
Could you please check and investigate? All screenshots are attached.

Kubernetes v1.15.3 (on-premise, kubespray)
Prometheus from helm chart stable/prometheus (9.1.1), node-exporter image prom/node-exporter:v0.18.0

[SergeiSporyshev]

Hi!
It's look like Prometheus cann't scrape metrics from kubelet (/metrics/cadvisor)

[amravyan]

In case you have Prometheus installed from prometheus chart (not prometheus operator), you have to add label replace rule listed below to two jobs: kubernetes-nodes and kubernetes-nodes-cadvisor.

          - source_labels: [__meta_kubernetes_node_name]
            action: replace
            target_label: node