Skip to content
This repository has been archived by the owner on Feb 22, 2022. It is now read-only.

cAdvisor scrape target for kubernetes v1.7+ #2700

Merged
merged 4 commits into from Jan 4, 2018
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion stable/prometheus/Chart.yaml
@@ -1,5 +1,5 @@
name: prometheus
version: 4.6.13
version: 4.6.14
description: Prometheus is a monitoring system and time series database.
home: https://prometheus.io/
icon: https://raw.githubusercontent.com/prometheus/prometheus.github.io/master/assets/prometheus_logo-cb55bb5c346.png
Expand Down
37 changes: 37 additions & 0 deletions stable/prometheus/values.yaml
Expand Up @@ -684,6 +684,43 @@ serverFiles:
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics


Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- job_name: 'kubernetes-nodes-cadvisor'

# Default to scraping over https. If required, just disable this or change to
# `http`.
scheme: https

# This TLS & bearer token file config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# <kubernetes_sd_config>.
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
# If your node certificates are self-signed or use a different CA to the
# master CA, then disable certificate verification below. Note that
# certificate verification is an integral part of a secure infrastructure
# so this should only be disabled in a controlled environment. You can
# disable certificate verification by uncommenting the line below.
#
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

kubernetes_sd_configs:
- role: node

relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}:4194/proxy/metrics
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed here that you're using the endpoint for 1.7.0 - 1.7.2.

According to the documentation added in prometheus/prometheus#2918, I believe that the configuration for 1.7.3+ should use replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor, no?

Copy link
Collaborator Author

@Miouge1 Miouge1 Nov 30, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The output from ${1}:4194/proxy/metrics and ${1}/proxy/metrics/cadvisor are different here is an example from a 1.7.3 cluster:

${1}:4194/proxy/metrics

container_cpu_cfs_periods_total{container_name="REDACTED",id="/kubepods/podc79ea696-d5a2-11e7-b0d7-fa163e5e1340/a6e2c7f17a46e1037f69f16cdd8cad3f970172a2f80d4570076be3c525cf2a7b",image="REDACTED",name="k8s_REDACTED",namespace="default",pod_name="REDACTED"} 2671

${1}/proxy/metrics/cadvisor

container_cpu_cfs_periods_total{id="/kubepods/podc79ea696-d5a2-11e7-b0d7-fa163e5e1340"} 2706

Empirical evidence shows that ${1}:4194/proxy/metrics has a lot more labels available than ${1}/proxy/metrics/cadvisor, that's why I went with this replacement config.

Copy link
Collaborator

@simonswine simonswine Dec 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With kubeadm (here on 1.8) the variant with 4194 fails as it is not enabled anymore. (See PR https://github.com/kubernetes/release/pull/356/files). Furthermore I think it is a bad practice to use an non encrypted port for that. I think the we should go with the metrics/cadvisor for the every Kubernetes > 1.7 through kubelet's endpoints

At least on 1.8 I see the full labels and that port and it works just fine with a default kubeadm deployment...


# Scrape config for service endpoints.
#
# The relabeling allows the actual service scrape endpoint to be configured
Expand Down