New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cAdvisor scrape target for kubernetes v1.7+ #2700

Merged
merged 4 commits into from Jan 4, 2018

Conversation

Projects
None yet
6 participants
@Miouge1
Copy link
Collaborator

Miouge1 commented Nov 9, 2017

Since Kubernetes v1.7 the cAdvisor /metrics is on a different port 4194 and needs an extra Prometheus job.

@Miouge1 Miouge1 force-pushed the Miouge1:prometheus-cadvisor branch to d438fa6 Nov 9, 2017

@Miouge1

This comment has been minimized.

Copy link
Collaborator

Miouge1 commented Nov 10, 2017

@mgoodness can you have a look at this?

- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}:4194/proxy/metrics

This comment has been minimized.

@nrmitchi

nrmitchi Nov 29, 2017

Contributor

I noticed here that you're using the endpoint for 1.7.0 - 1.7.2.

According to the documentation added in prometheus/prometheus#2918, I believe that the configuration for 1.7.3+ should use replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor, no?

This comment has been minimized.

@Miouge1

Miouge1 Nov 30, 2017

Collaborator

The output from ${1}:4194/proxy/metrics and ${1}/proxy/metrics/cadvisor are different here is an example from a 1.7.3 cluster:

${1}:4194/proxy/metrics

container_cpu_cfs_periods_total{container_name="REDACTED",id="/kubepods/podc79ea696-d5a2-11e7-b0d7-fa163e5e1340/a6e2c7f17a46e1037f69f16cdd8cad3f970172a2f80d4570076be3c525cf2a7b",image="REDACTED",name="k8s_REDACTED",namespace="default",pod_name="REDACTED"} 2671

${1}/proxy/metrics/cadvisor

container_cpu_cfs_periods_total{id="/kubepods/podc79ea696-d5a2-11e7-b0d7-fa163e5e1340"} 2706

Empirical evidence shows that ${1}:4194/proxy/metrics has a lot more labels available than ${1}/proxy/metrics/cadvisor, that's why I went with this replacement config.

This comment has been minimized.

@simonswine

simonswine Dec 5, 2017

Collaborator

With kubeadm (here on 1.8) the variant with 4194 fails as it is not enabled anymore. (See PR https://github.com/kubernetes/release/pull/356/files). Furthermore I think it is a bad practice to use an non encrypted port for that. I think the we should go with the metrics/cadvisor for the every Kubernetes > 1.7 through kubelet's endpoints

At least on 1.8 I see the full labels and that port and it works just fine with a default kubeadm deployment...

@Miouge1

This comment has been minimized.

Copy link
Collaborator

Miouge1 commented Nov 30, 2017

Resolving merge conflicts in the last commit.

@simonswine
Copy link
Collaborator

simonswine left a comment

Thank you for that update, had hacked that one or two times already it was about the time it gets done in the upstream helm chart 😆

- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}:4194/proxy/metrics

This comment has been minimized.

@simonswine

simonswine Dec 5, 2017

Collaborator

With kubeadm (here on 1.8) the variant with 4194 fails as it is not enabled anymore. (See PR https://github.com/kubernetes/release/pull/356/files). Furthermore I think it is a bad practice to use an non encrypted port for that. I think the we should go with the metrics/cadvisor for the every Kubernetes > 1.7 through kubelet's endpoints

At least on 1.8 I see the full labels and that port and it works just fine with a default kubeadm deployment...

@@ -684,6 +684,43 @@ serverFiles:
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
@simonswine

This comment has been minimized.

Copy link
Collaborator

simonswine commented Dec 5, 2017

/ok-to-test

@viglesiasce

This comment has been minimized.

Copy link
Contributor

viglesiasce commented Jan 4, 2018

/assign

@viglesiasce

This comment has been minimized.

Copy link
Contributor

viglesiasce commented Jan 4, 2018

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Jan 4, 2018

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 4, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Miouge1, viglesiasce

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 4f45dbc into helm:master Jan 4, 2018

3 checks passed

ci/circleci Your tests passed on CircleCI!
Details
cla/linuxfoundation viglesiasce authorized
Details
pull-charts-e2e Jenkins job succeeded.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment