New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes 1.7.0 requires cAdvisor changes #2916

Closed
unixwitch opened this Issue Jul 7, 2017 · 14 comments

Comments

Projects
None yet
8 participants
@unixwitch
Copy link
Contributor

unixwitch commented Jul 7, 2017

In Kubernetes 1.7.0, Kubelet no longer reports cAdvisor metrics on its /metrics endpoint; you have to scrape cAdvisor instead, on port 4194. This is apparently intentional, and means the provided kubernetes_sd_configs example doesn't work any more. (Well, it works, but it doesn't return container metrics, which makes it rather less useful than it should be.)

We observed this with an old config that looks like this:

- job_name: 'kubernetes-nodes-torchbox-staging'
[...]

  kubernetes_sd_configs:
  - role: node
    api_server: "https://apiserver.whatever:8443"
    [...]

  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)

Because kubernetes_sd_configs picks up the Kubelet port from the Node definition in apiserver, which is 10250 for us, it doesn't scrape port 4194 and therefore doesn't get the cAdvisor metrics.

We fixed this by adding a separate job to scrape port 4194:

- job_name: 'kubernetes-cadvisor-torchbox-staging'
  kubernetes_sd_configs:
  - role: node
    [...]

  relabel_configs:
  - source_labels: [__meta_kubernetes_node_address_InternalIP]
    target_label: __address__
    regex: (.*)
    replacement: $1:4194

However, looking at the current version of the example, it's now scraping Kubelet using the apiserver proxy. I assume this was changed to allow scraping nodes without requiring direct access to internal IP addresses. Unfortunately I don't know how to fix this to scrape cAdvisor as well, or if that's even possible. Obvious endpoints like /proxy/cadvisor don't work and documentation on the proxy endpoints seems hard to find.

In any case, the example config should probably be updated to scrape the new endpoint somehow.

(ref: kubernetes/kubernetes#48483)

@grobie

This comment has been minimized.

Copy link
Member

grobie commented Jul 7, 2017

I'd vote to start splitting the kubernetes example file by version as I know many companies who are (several) minor versions behind.

@unixwitch

This comment has been minimized.

Copy link
Contributor Author

unixwitch commented Jul 7, 2017

Okay, the cAdvisor endpoint is /api/v1/proxy/nodes/<node>:4194/metrics. So the example config could look like this:

- job_name: 'kubernetes-cadvisor'

  scheme: https
  
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  kubernetes_sd_configs:
  - role: node
  
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - target_label: __address__
    replacement: kubernetes.default.svc:443
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/proxy/nodes/${1}:4194/metrics

Should I open a PR, or would that be pointless if this will be rewritten anyway?

@juliusv

This comment has been minimized.

Copy link
Member

juliusv commented Jul 7, 2017

@unixwitch I think @grobie's proposal makes sense to split this up into different configs for different Kubernetes versions (we can start with 1.6 vs. 1.7), but otherwise I think a PR sounds great.

@unixwitch

This comment has been minimized.

Copy link
Contributor Author

unixwitch commented Jul 7, 2017

PR at #2918

@allistera

This comment has been minimized.

Copy link

allistera commented Jul 26, 2017

Does this require a change to the ClusterRole when rbac is enabled?

Changing to use /api/v1/proxy/nodes/<node>:4194/metrics starts to return a 403 - Forbidden from the nodes.

My ClusterRole is the same one from documentation/examples/rbac-setup.yml:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
@JorritSalverda

This comment has been minimized.

Copy link
Contributor

JorritSalverda commented Aug 1, 2017

I've just fixed the same issue by moving the /proxy/ part of the path behind instead of in front of the node name, like so:

-        replacement: /api/v1/proxy/nodes/${1}:4194/metrics
+        replacement: /api/v1/nodes/${1}:4194/proxy/metrics

This makes the - nodes/proxy resource in the ClusterRole apply to the request and let it through.

It might make sense to leave the kubernetes-nodes job as is without the cadvisor port and add the following separate job to scrape the cadvisor metrics:

    - job_name: 'kubernetes-cadvisor'

      scheme: https

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}:4194/proxy/metrics
@JorritSalverda

This comment has been minimized.

Copy link
Contributor

JorritSalverda commented Aug 1, 2017

Having both does lead to duplicate timeline series for kubernetes < 1.7 so make sure to either leave one of them out of your config or include the job in your alerts, rules and graphs. There's interesting info coming from the node itself though, so you probably want to keep scraping those as well from 1.7 onwards.

@unixwitch

This comment has been minimized.

Copy link
Contributor Author

unixwitch commented Aug 1, 2017

It might make sense to leave the kubernetes-nodes job as is without the cadvisor port and add the following separate job to scrape the cadvisor metrics

This is what the merged PR does, although it doesn't use the cadvisor port since in 1.7.3+ there is a new endpoint for this on the Kubelet metrics port (/metrics/cadvisor). See #2918 for the details.

@brancz

This comment has been minimized.

Copy link
Member

brancz commented Aug 1, 2017

The separate endpoint on the 4194 port has and will continue to exist though, so I think it's the common denominator in all setups, therefore the current canonical endpoint.

@unixwitch

This comment has been minimized.

Copy link
Contributor Author

unixwitch commented Aug 1, 2017

I don't think the example configuration should use the cadvisor (port 4191) endpoint. For one thing kubeadm now disables it by default (in 1.7+, I think), so it won't work in any cluster created by kubeadm. We also disable it in our own clusters as we prefer the encrypted and authenticated Kubelet endpoint.

By contrast the Kubelet endpoint (/metrics/cadvisor) is available in all clusters by default.

@brancz

This comment has been minimized.

Copy link
Member

brancz commented Aug 1, 2017

By contrast the Kubelet endpoint (/metrics/cadvisor) is available in all clusters by default.

This is only true for 1.7.3+ clusters.

But I understand your point. As this is just an example, I'd suggest we move this example to use the /metrics/cadvisor endpoint once it's been established enough (I'd say towards the 1.8 release).

@matthiasr

This comment has been minimized.

Copy link
Contributor

matthiasr commented Aug 2, 2017

@matthiasr

This comment has been minimized.

Copy link
Contributor

matthiasr commented Aug 3, 2017

1.7.3 is out now. the PR for this issue (#2918) is already merged, and the example configuration is very explicit about the different needs for different versions. I think we can close this issue.

@brancz

This comment has been minimized.

Copy link
Member

brancz commented Aug 3, 2017

Agreed @matthiasr. Thanks everyone!

@brancz brancz closed this Aug 3, 2017

dlebrero added a commit to akvo/akvo-platform that referenced this issue Dec 28, 2017

Add cAdvisor scraping on top of the default ones
The default configuration that comes with the Prometheus Helm chart does not scrapes the cAdvisor stats from cluster.

Added config as per prometheus/prometheus#2916 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment