Skip to content

Connected to cluster, but do not get CPU, Pod or Memory Usage #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sinkr opened this issue Nov 1, 2019 · 13 comments
Closed

Connected to cluster, but do not get CPU, Pod or Memory Usage #14

sinkr opened this issue Nov 1, 2019 · 13 comments

Comments

@sinkr
Copy link

sinkr commented Nov 1, 2019

v1.1.1 plugin
Grafana 6.4.3
Connected via token to the cluster, we do get statistics, just not the above-mentioned; also connected to Prometheus backend data source.

Hello,

Any idea why the gauges and the underlying values below aren't being populated for each node (See the NaN%s).

image

Great plugin, and thanks in advance for your help!

@SergeiSporyshev
Copy link
Collaborator

Hello!
Thank you for issue.
Check status of your kubelet exporter in prometheus and labels

https://<your_cluster_ip>:10250/metrics
https://<your_cluster_ip>:10250/metrics/cadvisor

@sinkr
Copy link
Author

sinkr commented Nov 1, 2019 via email

@sinkr
Copy link
Author

sinkr commented Nov 1, 2019

Sergei,

When I click on the chart that has no data and go to "explore," I get this query:

(sum(rate(container_cpu_usage_seconds_total{node=~"ip-10-1-56-122.ec2.internal"}[1m])) / sum(kube_node_status_allocatable_cpu_cores{node=~"ip-10-1-56-122.ec2.internal"})) * 100

When I copy and paste this into my Prometheus query window, I get no datapoints. Do you think I have a broken Prometheus install, despite many other metrics being populated?

@SergeiSporyshev
Copy link
Collaborator

Check your relabel configs in Prometheus

#11

@SergeiSporyshev
Copy link
Collaborator

Right now, what I'm doing is using the token to communicate with the
cluster. Should I generate the SSL credentials instead? Would that fix
anything?

Which version of plugin do you use?

@sinkr
Copy link
Author

sinkr commented Nov 4, 2019 via email

@SergeiSporyshev
Copy link
Collaborator

Hi!

This is part of my Prometheus' config

`- job_name: monitoring/monitoring-prometheus-oper-kubelet/1
honor_labels: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics/cadvisor
scheme: http
kubernetes_sd_configs:

  • role: endpoints
    namespaces:
    names:

    • kube-system
      relabel_configs:
  • source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kubelet
    replacement: $1
    action: keep

  • source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep

  • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace

  • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace

  • source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace

  • source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace

  • source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace

  • source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace

  • source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace

  • separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace

  • job_name: monitoring/monitoring-prometheus-oper-kubelet/0
    honor_labels: true
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    kubernetes_sd_configs:

    • role: endpoints
      namespaces:
      names:
      • kube-system
        relabel_configs:
    • source_labels: [__meta_kubernetes_service_label_k8s_app]
      separator: ;
      regex: kubelet
      replacement: $1
      action: keep
    • source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: http-metrics
      replacement: $1
      action: keep
    • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Node;(.*)
      target_label: node
      replacement: ${1}
      action: replace
    • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Pod;(.*)
      target_label: pod
      replacement: ${1}
      action: replace
    • source_labels: [__meta_kubernetes_namespace]
      separator: ;
      regex: (.*)
      target_label: namespace
      replacement: $1
      action: replace
    • source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: (.*)
      target_label: service
      replacement: $1
      action: replace
    • source_labels: [__meta_kubernetes_pod_name]
      separator: ;
      regex: (.*)
      target_label: pod
      replacement: $1
      action: replace
    • source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: (.*)
      target_label: job
      replacement: ${1}
      action: replace
    • source_labels: [__meta_kubernetes_service_label_k8s_app]
      separator: ;
      regex: (.+)
      target_label: job
      replacement: ${1}
      action: replace
    • separator: ;
      regex: (.*)
      target_label: endpoint
      replacement: http-metrics
      action: replace`

And try to install Prometheus via Prometheus-operator (https://github.com/helm/charts/tree/master/stable/prometheus-operator)

@sinkr
Copy link
Author

sinkr commented Nov 5, 2019 via email

@SergeiSporyshev
Copy link
Collaborator

Hi @sinkr !
What version of node-exporter and kube-state-metrics you use?
I don't see this targets in your config

@sinkr
Copy link
Author

sinkr commented Nov 12, 2019 via email

@SergeiSporyshev
Copy link
Collaborator

Hi @sinkr!
We had the same problem during tests.
Can you check it by new release (via git pull) - https://github.com/devopsprodigy/kubegraf/releases/tag/v1.2.0.5 ?

@sinkr
Copy link
Author

sinkr commented Nov 25, 2019 via email

@SergeiSporyshev
Copy link
Collaborator

Yeah!
Thank you for this issue again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants