-
Notifications
You must be signed in to change notification settings - Fork 44
Connected to cluster, but do not get CPU, Pod or Memory Usage #14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Hello! https://<your_cluster_ip>:10250/metrics |
Hi Sergei,
Thank you for the response. I can get to https://cluster_ip:10250/metrics
and /metrics/cadvisor, however, I'm assuming I'm getting an "Unauthorized"
at the kubegraf level.
Right now, what I'm doing is using the token to communicate with the
cluster. Should I generate the SSL credentials instead? Would that fix
anything?
…On Fri, Nov 1, 2019 at 8:47 AM SergeiSporyshev ***@***.***> wrote:
Hello!
Thank you for issue.
Check status of your kubelet exporter in prometheus and labels
https://<your_cluster_ip>:10250/metrics
https://<your_cluster_ip>:10250/metrics/cadvisor
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#14?email_source=notifications&email_token=AAFKANR6NWEW4C73I32MT4LQRQQPHA5CNFSM4JHVYMM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC22OSQ#issuecomment-548775754>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFKANQ3OV4WYORD537PEBDQRQQPHANCNFSM4JHVYMMQ>
.
|
Sergei, When I click on the chart that has no data and go to "explore," I get this query:
When I copy and paste this into my Prometheus query window, I get no datapoints. Do you think I have a broken Prometheus install, despite many other metrics being populated? |
Check your relabel configs in Prometheus |
Which version of plugin do you use? |
Hi Sergei,
Thank you again for the response. I added the relabel configs and
restarted my Prometheus server, however, the chart still is not picking up
node (instance works in its place).
Is there anything else I should do?
…On Sat, Nov 2, 2019 at 12:08 AM SergeiSporyshev ***@***.***> wrote:
Check your relabel configs in Prometheus
#11 <#11>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#14?email_source=notifications&email_token=AAFKANRLJ4OAX67BNOWY6WDQRT4M7A5CNFSM4JHVYMM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC4TI5I#issuecomment-549008501>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFKANVQOB4KHDFFRVDYURLQRT4M7ANCNFSM4JHVYMMQ>
.
|
Hi! This is part of my Prometheus' config `- job_name: monitoring/monitoring-prometheus-oper-kubelet/1
And try to install Prometheus via Prometheus-operator (https://github.com/helm/charts/tree/master/stable/prometheus-operator) |
Hi Sergei,
I did install Prometheus via helm, and when I went to install the operator
component, I was told that it was already installed.
I did add the relabel configs as per the previous issue and here's what my
config looks like--sorry for all the Q's--but do you see anything here?
There's a lot to look at, but it seems like what you pasted I have none of
in my config file:
global:
scrape_interval: 1m
scrape_timeout: 10s
evaluation_interval: 1m
alerting:
alertmanagers:
- kubernetes_sd_configs:
- role: pod
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
scheme: http
timeout: 10s
api_version: v1
relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: prometheus
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_pod_label_app]
separator: ;
regex: prometheus
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_pod_label_component]
separator: ;
regex: alertmanager
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_pod_container_port_number]
separator: ;
regex: null
replacement: $1
action: drop
rule_files:
- /etc/config/rules
- /etc/config/alerts
scrape_configs:
- job_name: prometheus
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
static_configs:
- targets:
- localhost:9090
- job_name: kubernetes-apiservers
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
kubernetes_sd_configs:
- role: endpoints
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
relabel_configs:
- source_labels: [__meta_kubernetes_namespace,
__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
separator: ;
regex: default;kubernetes;https
replacement: $1
action: keep
- job_name: kubernetes-nodes
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
kubernetes_sd_configs:
- role: node
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
relabel_configs:
- separator: ;
regex: __meta_kubernetes_node_label_(.+)
replacement: $1
action: labelmap
- separator: ;
regex: (.*)
target_label: __address__
replacement: kubernetes.default.svc:443
action: replace
- source_labels: [__meta_kubernetes_node_name]
separator: ;
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/$1/proxy/metrics
action: replace
- separator: ;
regex: __meta_kubernetes_node_label_kubernetes_io_hostname
replacement: node
action: labelmap
- job_name: kubernetes-nodes-cadvisor
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
kubernetes_sd_configs:
- role: node
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
relabel_configs:
- separator: ;
regex: __meta_kubernetes_node_label_(.+)
replacement: $1
action: labelmap
- separator: ;
regex: (.*)
target_label: __address__
replacement: kubernetes.default.svc:443
action: replace
- source_labels: [__meta_kubernetes_node_name]
separator: ;
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
action: replace
- separator: ;
regex: __meta_kubernetes_node_label_kubernetes_io_hostname
replacement: node
action: labelmap
- job_name: kubernetes-service-endpoints
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
separator: ;
regex: "true"
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
separator: ;
regex: (https?)
target_label: __scheme__
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
separator: ;
regex: (.+)
target_label: __metrics_path__
replacement: $1
action: replace
- source_labels: [__address__,
__meta_kubernetes_service_annotation_prometheus_io_port]
separator: ;
regex: ([^:]+)(?::\d+)?;(\d+)
target_label: __address__
replacement: $1:$2
action: replace
- separator: ;
regex: __meta_kubernetes_service_label_(.+)
replacement: $1
action: labelmap
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: kubernetes_namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: kubernetes_name
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_node_name]
separator: ;
regex: (.*)
target_label: kubernetes_node
replacement: $1
action: replace
- job_name: prometheus-pushgateway
honor_labels: true
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
kubernetes_sd_configs:
- role: service
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
separator: ;
regex: pushgateway
replacement: $1
action: keep
- job_name: kubernetes-services
honor_timestamps: true
params:
module:
- http_2xx
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /probe
scheme: http
kubernetes_sd_configs:
- role: service
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
separator: ;
regex: "true"
replacement: $1
action: keep
- source_labels: [__address__]
separator: ;
regex: (.*)
target_label: __param_target
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: __address__
replacement: blackbox
action: replace
- source_labels: [__param_target]
separator: ;
regex: (.*)
target_label: instance
replacement: $1
action: replace
- separator: ;
regex: __meta_kubernetes_service_label_(.+)
replacement: $1
action: labelmap
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: kubernetes_namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: kubernetes_name
replacement: $1
action: replace
- job_name: kubernetes-pods
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
separator: ;
regex: "true"
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
separator: ;
regex: (.+)
target_label: __metrics_path__
replacement: $1
action: replace
- source_labels: [__address__,
__meta_kubernetes_pod_annotation_prometheus_io_port]
separator: ;
regex: ([^:]+)(?::\d+)?;(\d+)
target_label: __address__
replacement: $1:$2
action: replace
- separator: ;
regex: __meta_kubernetes_pod_label_(.+)
replacement: $1
action: labelmap
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: kubernetes_namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: kubernetes_pod_name
replacement: $1
action: replace
…On Mon, Nov 4, 2019 at 11:17 PM SergeiSporyshev ***@***.***> wrote:
Hi!
This is part of my Prometheus' config
`- job_name: monitoring/monitoring-prometheus-oper-kubelet/1
honor_labels: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics/cadvisor
scheme: http
kubernetes_sd_configs:
-
role: endpoints
namespaces:
names:
- kube-system
relabel_configs:
-
source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: kubelet
replacement: $1
action: keep
-
source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: http-metrics
replacement: $1
action: keep
-
source_labels: [__meta_kubernetes_endpoint_address_target_kind,
__meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
-
source_labels: [__meta_kubernetes_endpoint_address_target_kind,
__meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
-
source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
-
source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
-
source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
-
source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
-
source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: (.+)
target_label: job
replacement: ${1}
action: replace
-
separator: ;
regex: (.*)
target_label: endpoint
replacement: http-metrics
action: replace
-
job_name: monitoring/monitoring-prometheus-oper-kubelet/0
honor_labels: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- kube-system
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: kubelet
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: http-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind,
__meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind,
__meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: (.+)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: http-metrics
action: replace`
And try to install Prometheus via Prometheus-operator (
https://github.com/helm/charts/tree/master/stable/prometheus-operator)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#14?email_source=notifications&email_token=AAFKANU2ZGAMZTMNRG5V5JLQSDXVPA5CNFSM4JHVYMM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDBRNBA#issuecomment-549656196>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFKANUE4A4JZEMXNJZM6S3QSDXVPANCNFSM4JHVYMMQ>
.
|
Hi @sinkr ! |
Hi Sergei,
I am running 0.18.0 node-exporter and 1.6.0 of
prometheus-kube-state-metrics.
I can make the the CPU gauge populate by changing the query to look like
this:
(sum(rate(container_cpu_usage_seconds_total{instance="$node"}[15m])) /
sum(kube_node_status_allocatable_cpu_cores{node="$node"})) * 100
The material change is instance="$node"; just an FYI.
…On Tue, Nov 12, 2019 at 7:45 AM SergeiSporyshev ***@***.***> wrote:
Hi @sinkr <https://github.com/sinkr> !
What version of node-exporter and kube-state-metrics you use?
I don't see this targets in your config
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#14?email_source=notifications&email_token=AAFKANTFOGDSPMQATMFA6STQTKQPHA5CNFSM4JHVYMM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED2ECFQ#issuecomment-552878358>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFKANROCLNSFMCKO5ECQ53QTKQPHANCNFSM4JHVYMMQ>
.
|
Hi @sinkr! |
Hi Sergei,
Yes, this fixed it (v1.2.0.5)!
Thank you.
…On Sat, Nov 23, 2019 at 6:40 AM SergeiSporyshev ***@***.***> wrote:
Hi @sinkr <https://github.com/sinkr>!
We had the same problem during tests.
Can you check it by new release (via git pull) -
https://github.com/devopsprodigy/kubegraf/releases/tag/v1.2.0.5 ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#14?email_source=notifications&email_token=AAFKANVZJTZKDRVRB44ACQTQVEJDNA5CNFSM4JHVYMM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE7TMKY#issuecomment-557790763>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFKANV2NSNK7AN5B435IEDQVEJDNANCNFSM4JHVYMMQ>
.
|
Yeah! |
v1.1.1 plugin
Grafana 6.4.3
Connected via token to the cluster, we do get statistics, just not the above-mentioned; also connected to Prometheus backend data source.
Hello,
Any idea why the gauges and the underlying values below aren't being populated for each node (See the NaN%s).
Great plugin, and thanks in advance for your help!
The text was updated successfully, but these errors were encountered: