Connected to cluster, but do not get CPU, Pod or Memory Usage

[sinkr]

v1.1.1 plugin
Grafana 6.4.3
Connected via token to the cluster, we do get statistics, just not the above-mentioned; also connected to Prometheus backend data source.

Hello,

Any idea why the gauges and the underlying values below aren't being populated for each node (See the NaN%s).

Great plugin, and thanks in advance for your help!

[SergeiSporyshev]

Hello!
Thank you for issue.
Check status of your kubelet exporter in prometheus and labels

https://<your_cluster_ip>:10250/metrics
https://<your_cluster_ip>:10250/metrics/cadvisor

[sinkr]

Hi Sergei, Thank you for the response. I can get to https://cluster_ip:10250/metrics and /metrics/cadvisor, however, I'm assuming I'm getting an "Unauthorized" at the kubegraf level. Right now, what I'm doing is using the token to communicate with the cluster. Should I generate the SSL credentials instead? Would that fix anything?

…

On Fri, Nov 1, 2019 at 8:47 AM SergeiSporyshev ***@***.***> wrote: Hello! Thank you for issue. Check status of your kubelet exporter in prometheus and labels https://<your_cluster_ip>:10250/metrics https://<your_cluster_ip>:10250/metrics/cadvisor — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14?email_source=notifications&email_token=AAFKANR6NWEW4C73I32MT4LQRQQPHA5CNFSM4JHVYMM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC22OSQ#issuecomment-548775754>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFKANQ3OV4WYORD537PEBDQRQQPHANCNFSM4JHVYMMQ> .

[sinkr]

Sergei,

When I click on the chart that has no data and go to "explore," I get this query:

(sum(rate(container_cpu_usage_seconds_total{node=~"ip-10-1-56-122.ec2.internal"}[1m])) / sum(kube_node_status_allocatable_cpu_cores{node=~"ip-10-1-56-122.ec2.internal"})) * 100

When I copy and paste this into my Prometheus query window, I get no datapoints. Do you think I have a broken Prometheus install, despite many other metrics being populated?

[SergeiSporyshev]

Check your relabel configs in Prometheus

#11

[SergeiSporyshev]

Right now, what I'm doing is using the token to communicate with the
cluster. Should I generate the SSL credentials instead? Would that fix
anything?

Which version of plugin do you use?

[sinkr]

Hi Sergei, Thank you again for the response. I added the relabel configs and restarted my Prometheus server, however, the chart still is not picking up node (instance works in its place). Is there anything else I should do?

…

On Sat, Nov 2, 2019 at 12:08 AM SergeiSporyshev ***@***.***> wrote: Check your relabel configs in Prometheus #11 <#11> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14?email_source=notifications&email_token=AAFKANRLJ4OAX67BNOWY6WDQRT4M7A5CNFSM4JHVYMM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC4TI5I#issuecomment-549008501>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFKANVQOB4KHDFFRVDYURLQRT4M7ANCNFSM4JHVYMMQ> .

[SergeiSporyshev]

Hi!

This is part of my Prometheus' config

`- job_name: monitoring/monitoring-prometheus-oper-kubelet/1
honor_labels: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics/cadvisor
scheme: http
kubernetes_sd_configs:

• role: endpoints
  namespaces:
  names:

  • kube-system
    relabel_configs:

• source_labels: [__meta_kubernetes_service_label_k8s_app]
  separator: ;
  regex: kubelet
  replacement: $1
  action: keep

• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: http-metrics
  replacement: $1
  action: keep

• source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
  separator: ;
  regex: Node;(.*)
  target_label: node
  replacement: ${1}
  action: replace

• source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
  separator: ;
  regex: Pod;(.*)
  target_label: pod
  replacement: ${1}
  action: replace

• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace

• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace

• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace

• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace

• source_labels: [__meta_kubernetes_service_label_k8s_app]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace

• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: http-metrics
  action: replace

• job_name: monitoring/monitoring-prometheus-oper-kubelet/0
  honor_labels: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:

  • role: endpoints
    namespaces:
    names:
    • kube-system
      relabel_configs:
  • source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kubelet
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  • source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  • source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  • separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace`

And try to install Prometheus via Prometheus-operator (https://github.com/helm/charts/tree/master/stable/prometheus-operator)

[sinkr]

Hi Sergei, I did install Prometheus via helm, and when I went to install the operator component, I was told that it was already installed. I did add the relabel configs as per the previous issue and here's what my config looks like--sorry for all the Q's--but do you see anything here? There's a lot to look at, but it seems like what you pasted I have none of in my config file: global: scrape_interval: 1m scrape_timeout: 10s evaluation_interval: 1m alerting: alertmanagers: - kubernetes_sd_configs: - role: pod bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: false scheme: http timeout: 10s api_version: v1 relabel_configs: - source_labels: [__meta_kubernetes_namespace] separator: ; regex: prometheus replacement: $1 action: keep - source_labels: [__meta_kubernetes_pod_label_app] separator: ; regex: prometheus replacement: $1 action: keep - source_labels: [__meta_kubernetes_pod_label_component] separator: ; regex: alertmanager replacement: $1 action: keep - source_labels: [__meta_kubernetes_pod_container_port_number] separator: ; regex: null replacement: $1 action: drop rule_files: - /etc/config/rules - /etc/config/alerts scrape_configs: - job_name: prometheus honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: http static_configs: - targets: - localhost:9090 - job_name: kubernetes-apiservers honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: https kubernetes_sd_configs: - role: endpoints bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] separator: ; regex: default;kubernetes;https replacement: $1 action: keep - job_name: kubernetes-nodes honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: https kubernetes_sd_configs: - role: node bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - separator: ; regex: __meta_kubernetes_node_label_(.+) replacement: $1 action: labelmap - separator: ; regex: (.*) target_label: __address__ replacement: kubernetes.default.svc:443 action: replace - source_labels: [__meta_kubernetes_node_name] separator: ; regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/$1/proxy/metrics action: replace - separator: ; regex: __meta_kubernetes_node_label_kubernetes_io_hostname replacement: node action: labelmap - job_name: kubernetes-nodes-cadvisor honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: https kubernetes_sd_configs: - role: node bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - separator: ; regex: __meta_kubernetes_node_label_(.+) replacement: $1 action: labelmap - separator: ; regex: (.*) target_label: __address__ replacement: kubernetes.default.svc:443 action: replace - source_labels: [__meta_kubernetes_node_name] separator: ; regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor action: replace - separator: ; regex: __meta_kubernetes_node_label_kubernetes_io_hostname replacement: node action: labelmap - job_name: kubernetes-service-endpoints honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: http kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] separator: ; regex: "true" replacement: $1 action: keep - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] separator: ; regex: (https?) target_label: __scheme__ replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] separator: ; regex: (.+) target_label: __metrics_path__ replacement: $1 action: replace - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] separator: ; regex: ([^:]+)(?::\d+)?;(\d+) target_label: __address__ replacement: $1:$2 action: replace - separator: ; regex: __meta_kubernetes_service_label_(.+) replacement: $1 action: labelmap - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: kubernetes_namespace replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: kubernetes_name replacement: $1 action: replace - source_labels: [__meta_kubernetes_pod_node_name] separator: ; regex: (.*) target_label: kubernetes_node replacement: $1 action: replace - job_name: prometheus-pushgateway honor_labels: true honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: http kubernetes_sd_configs: - role: service relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe] separator: ; regex: pushgateway replacement: $1 action: keep - job_name: kubernetes-services honor_timestamps: true params: module: - http_2xx scrape_interval: 1m scrape_timeout: 10s metrics_path: /probe scheme: http kubernetes_sd_configs: - role: service relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe] separator: ; regex: "true" replacement: $1 action: keep - source_labels: [__address__] separator: ; regex: (.*) target_label: __param_target replacement: $1 action: replace - separator: ; regex: (.*) target_label: __address__ replacement: blackbox action: replace - source_labels: [__param_target] separator: ; regex: (.*) target_label: instance replacement: $1 action: replace - separator: ; regex: __meta_kubernetes_service_label_(.+) replacement: $1 action: labelmap - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: kubernetes_namespace replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: kubernetes_name replacement: $1 action: replace - job_name: kubernetes-pods honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: http kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] separator: ; regex: "true" replacement: $1 action: keep - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] separator: ; regex: (.+) target_label: __metrics_path__ replacement: $1 action: replace - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] separator: ; regex: ([^:]+)(?::\d+)?;(\d+) target_label: __address__ replacement: $1:$2 action: replace - separator: ; regex: __meta_kubernetes_pod_label_(.+) replacement: $1 action: labelmap - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: kubernetes_namespace replacement: $1 action: replace - source_labels: [__meta_kubernetes_pod_name] separator: ; regex: (.*) target_label: kubernetes_pod_name replacement: $1 action: replace

…

On Mon, Nov 4, 2019 at 11:17 PM SergeiSporyshev ***@***.***> wrote: Hi! This is part of my Prometheus' config `- job_name: monitoring/monitoring-prometheus-oper-kubelet/1 honor_labels: true scrape_interval: 30s scrape_timeout: 10s metrics_path: /metrics/cadvisor scheme: http kubernetes_sd_configs: - role: endpoints namespaces: names: - kube-system relabel_configs: - source_labels: [__meta_kubernetes_service_label_k8s_app] separator: ; regex: kubelet replacement: $1 action: keep - source_labels: [__meta_kubernetes_endpoint_port_name] separator: ; regex: http-metrics replacement: $1 action: keep - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name] separator: ; regex: Node;(.*) target_label: node replacement: ${1} action: replace - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name] separator: ; regex: Pod;(.*) target_label: pod replacement: ${1} action: replace - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: namespace replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: service replacement: $1 action: replace - source_labels: [__meta_kubernetes_pod_name] separator: ; regex: (.*) target_label: pod replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: job replacement: ${1} action: replace - source_labels: [__meta_kubernetes_service_label_k8s_app] separator: ; regex: (.+) target_label: job replacement: ${1} action: replace - separator: ; regex: (.*) target_label: endpoint replacement: http-metrics action: replace - job_name: monitoring/monitoring-prometheus-oper-kubelet/0 honor_labels: true scrape_interval: 30s scrape_timeout: 10s metrics_path: /metrics scheme: http kubernetes_sd_configs: - role: endpoints namespaces: names: - kube-system relabel_configs: - source_labels: [__meta_kubernetes_service_label_k8s_app] separator: ; regex: kubelet replacement: $1 action: keep - source_labels: [__meta_kubernetes_endpoint_port_name] separator: ; regex: http-metrics replacement: $1 action: keep - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name] separator: ; regex: Node;(.*) target_label: node replacement: ${1} action: replace - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name] separator: ; regex: Pod;(.*) target_label: pod replacement: ${1} action: replace - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: namespace replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: service replacement: $1 action: replace - source_labels: [__meta_kubernetes_pod_name] separator: ; regex: (.*) target_label: pod replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: job replacement: ${1} action: replace - source_labels: [__meta_kubernetes_service_label_k8s_app] separator: ; regex: (.+) target_label: job replacement: ${1} action: replace - separator: ; regex: (.*) target_label: endpoint replacement: http-metrics action: replace` And try to install Prometheus via Prometheus-operator ( https://github.com/helm/charts/tree/master/stable/prometheus-operator) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14?email_source=notifications&email_token=AAFKANU2ZGAMZTMNRG5V5JLQSDXVPA5CNFSM4JHVYMM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDBRNBA#issuecomment-549656196>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFKANUE4A4JZEMXNJZM6S3QSDXVPANCNFSM4JHVYMMQ> .

[SergeiSporyshev]

Hi @sinkr !
What version of node-exporter and kube-state-metrics you use?
I don't see this targets in your config

[sinkr]

Hi Sergei, I am running 0.18.0 node-exporter and 1.6.0 of prometheus-kube-state-metrics. I can make the the CPU gauge populate by changing the query to look like this: (sum(rate(container_cpu_usage_seconds_total{instance="$node"}[15m])) / sum(kube_node_status_allocatable_cpu_cores{node="$node"})) * 100 The material change is instance="$node"; just an FYI.

…

On Tue, Nov 12, 2019 at 7:45 AM SergeiSporyshev ***@***.***> wrote: Hi @sinkr <https://github.com/sinkr> ! What version of node-exporter and kube-state-metrics you use? I don't see this targets in your config — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14?email_source=notifications&email_token=AAFKANTFOGDSPMQATMFA6STQTKQPHA5CNFSM4JHVYMM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED2ECFQ#issuecomment-552878358>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFKANROCLNSFMCKO5ECQ53QTKQPHANCNFSM4JHVYMMQ> .

[SergeiSporyshev]

Hi @sinkr!
We had the same problem during tests.
Can you check it by new release (via git pull) - https://github.com/devopsprodigy/kubegraf/releases/tag/v1.2.0.5 ?

[sinkr]

Hi Sergei, Yes, this fixed it (v1.2.0.5)! Thank you.

…

On Sat, Nov 23, 2019 at 6:40 AM SergeiSporyshev ***@***.***> wrote: Hi @sinkr <https://github.com/sinkr>! We had the same problem during tests. Can you check it by new release (via git pull) - https://github.com/devopsprodigy/kubegraf/releases/tag/v1.2.0.5 ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14?email_source=notifications&email_token=AAFKANVZJTZKDRVRB44ACQTQVEJDNA5CNFSM4JHVYMM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE7TMKY#issuecomment-557790763>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFKANV2NSNK7AN5B435IEDQVEJDNANCNFSM4JHVYMMQ> .

[SergeiSporyshev]

Yeah!
Thank you for this issue again!