Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connected to cluster, but do not get CPU, Pod or Memory Usage #14

Closed
sinkr opened this issue Nov 1, 2019 · 13 comments
Closed

Connected to cluster, but do not get CPU, Pod or Memory Usage #14

sinkr opened this issue Nov 1, 2019 · 13 comments

Comments

@sinkr
Copy link

@sinkr sinkr commented Nov 1, 2019

v1.1.1 plugin
Grafana 6.4.3
Connected via token to the cluster, we do get statistics, just not the above-mentioned; also connected to Prometheus backend data source.

Hello,

Any idea why the gauges and the underlying values below aren't being populated for each node (See the NaN%s).

image

Great plugin, and thanks in advance for your help!

@SergeiSporyshev

This comment has been minimized.

Copy link
Collaborator

@SergeiSporyshev SergeiSporyshev commented Nov 1, 2019

Hello!
Thank you for issue.
Check status of your kubelet exporter in prometheus and labels

https://<your_cluster_ip>:10250/metrics
https://<your_cluster_ip>:10250/metrics/cadvisor

@sinkr

This comment has been minimized.

Copy link
Author

@sinkr sinkr commented Nov 1, 2019

@sinkr

This comment has been minimized.

Copy link
Author

@sinkr sinkr commented Nov 1, 2019

Sergei,

When I click on the chart that has no data and go to "explore," I get this query:

(sum(rate(container_cpu_usage_seconds_total{node=~"ip-10-1-56-122.ec2.internal"}[1m])) / sum(kube_node_status_allocatable_cpu_cores{node=~"ip-10-1-56-122.ec2.internal"})) * 100

When I copy and paste this into my Prometheus query window, I get no datapoints. Do you think I have a broken Prometheus install, despite many other metrics being populated?

@SergeiSporyshev

This comment has been minimized.

Copy link
Collaborator

@SergeiSporyshev SergeiSporyshev commented Nov 2, 2019

Check your relabel configs in Prometheus

#11

@SergeiSporyshev

This comment has been minimized.

Copy link
Collaborator

@SergeiSporyshev SergeiSporyshev commented Nov 2, 2019

Right now, what I'm doing is using the token to communicate with the
cluster. Should I generate the SSL credentials instead? Would that fix
anything?

Which version of plugin do you use?

@sinkr

This comment has been minimized.

Copy link
Author

@sinkr sinkr commented Nov 4, 2019

@SergeiSporyshev

This comment has been minimized.

Copy link
Collaborator

@SergeiSporyshev SergeiSporyshev commented Nov 5, 2019

Hi!

This is part of my Prometheus' config

`- job_name: monitoring/monitoring-prometheus-oper-kubelet/1
honor_labels: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics/cadvisor
scheme: http
kubernetes_sd_configs:

  • role: endpoints
    namespaces:
    names:

    • kube-system
      relabel_configs:
  • source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kubelet
    replacement: $1
    action: keep

  • source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep

  • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace

  • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace

  • source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace

  • source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace

  • source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace

  • source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace

  • source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace

  • separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace

  • job_name: monitoring/monitoring-prometheus-oper-kubelet/0
    honor_labels: true
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    kubernetes_sd_configs:

    • role: endpoints
      namespaces:
      names:
      • kube-system
        relabel_configs:
    • source_labels: [__meta_kubernetes_service_label_k8s_app]
      separator: ;
      regex: kubelet
      replacement: $1
      action: keep
    • source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: http-metrics
      replacement: $1
      action: keep
    • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Node;(.*)
      target_label: node
      replacement: ${1}
      action: replace
    • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Pod;(.*)
      target_label: pod
      replacement: ${1}
      action: replace
    • source_labels: [__meta_kubernetes_namespace]
      separator: ;
      regex: (.*)
      target_label: namespace
      replacement: $1
      action: replace
    • source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: (.*)
      target_label: service
      replacement: $1
      action: replace
    • source_labels: [__meta_kubernetes_pod_name]
      separator: ;
      regex: (.*)
      target_label: pod
      replacement: $1
      action: replace
    • source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: (.*)
      target_label: job
      replacement: ${1}
      action: replace
    • source_labels: [__meta_kubernetes_service_label_k8s_app]
      separator: ;
      regex: (.+)
      target_label: job
      replacement: ${1}
      action: replace
    • separator: ;
      regex: (.*)
      target_label: endpoint
      replacement: http-metrics
      action: replace`

And try to install Prometheus via Prometheus-operator (https://github.com/helm/charts/tree/master/stable/prometheus-operator)

@sinkr

This comment has been minimized.

Copy link
Author

@sinkr sinkr commented Nov 5, 2019

@SergeiSporyshev

This comment has been minimized.

Copy link
Collaborator

@SergeiSporyshev SergeiSporyshev commented Nov 12, 2019

Hi @sinkr !
What version of node-exporter and kube-state-metrics you use?
I don't see this targets in your config

@sinkr

This comment has been minimized.

Copy link
Author

@sinkr sinkr commented Nov 12, 2019

@SergeiSporyshev

This comment has been minimized.

Copy link
Collaborator

@SergeiSporyshev SergeiSporyshev commented Nov 23, 2019

Hi @sinkr!
We had the same problem during tests.
Can you check it by new release (via git pull) - https://github.com/devopsprodigy/kubegraf/releases/tag/v1.2.0.5 ?

@sinkr

This comment has been minimized.

Copy link
Author

@sinkr sinkr commented Nov 25, 2019

@SergeiSporyshev

This comment has been minimized.

Copy link
Collaborator

@SergeiSporyshev SergeiSporyshev commented Nov 26, 2019

Yeah!
Thank you for this issue again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.