Kubernetes 1.7.0 requires cAdvisor changes

[unixwitch]

In Kubernetes 1.7.0, Kubelet no longer reports cAdvisor metrics on its /metrics endpoint; you have to scrape cAdvisor instead, on port 4194. This is apparently intentional, and means the provided kubernetes_sd_configs example doesn't work any more. (Well, it works, but it doesn't return container metrics, which makes it rather less useful than it should be.)

We observed this with an old config that looks like this:

- job_name: 'kubernetes-nodes-torchbox-staging'
[...]

  kubernetes_sd_configs:
  - role: node
    api_server: "https://apiserver.whatever:8443"
    [...]

  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)

Because kubernetes_sd_configs picks up the Kubelet port from the Node definition in apiserver, which is 10250 for us, it doesn't scrape port 4194 and therefore doesn't get the cAdvisor metrics.

We fixed this by adding a separate job to scrape port 4194:

- job_name: 'kubernetes-cadvisor-torchbox-staging'
  kubernetes_sd_configs:
  - role: node
    [...]

  relabel_configs:
  - source_labels: [__meta_kubernetes_node_address_InternalIP]
    target_label: __address__
    regex: (.*)
    replacement: $1:4194

However, looking at the current version of the example, it's now scraping Kubelet using the apiserver proxy. I assume this was changed to allow scraping nodes without requiring direct access to internal IP addresses. Unfortunately I don't know how to fix this to scrape cAdvisor as well, or if that's even possible. Obvious endpoints like /proxy/cadvisor don't work and documentation on the proxy endpoints seems hard to find.

In any case, the example config should probably be updated to scrape the new endpoint somehow.

(ref: kubernetes/kubernetes#48483)

[grobie]

I'd vote to start splitting the kubernetes example file by version as I know many companies who are (several) minor versions behind.

[unixwitch]

Okay, the cAdvisor endpoint is /api/v1/proxy/nodes/<node>:4194/metrics. So the example config could look like this:

- job_name: 'kubernetes-cadvisor'

  scheme: https
  
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  kubernetes_sd_configs:
  - role: node
  
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - target_label: __address__
    replacement: kubernetes.default.svc:443
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/proxy/nodes/${1}:4194/metrics

Should I open a PR, or would that be pointless if this will be rewritten anyway?

[juliusv]

@unixwitch I think @grobie's proposal makes sense to split this up into different configs for different Kubernetes versions (we can start with 1.6 vs. 1.7), but otherwise I think a PR sounds great.

[unixwitch]

PR at #2918

[allistera]

Does this require a change to the ClusterRole when rbac is enabled?

Changing to use /api/v1/proxy/nodes/<node>:4194/metrics starts to return a 403 - Forbidden from the nodes.

My ClusterRole is the same one from documentation/examples/rbac-setup.yml:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

[JorritSalverda]

I've just fixed the same issue by moving the /proxy/ part of the path behind instead of in front of the node name, like so:

-        replacement: /api/v1/proxy/nodes/${1}:4194/metrics
+        replacement: /api/v1/nodes/${1}:4194/proxy/metrics

This makes the - nodes/proxy resource in the ClusterRole apply to the request and let it through.

It might make sense to leave the kubernetes-nodes job as is without the cadvisor port and add the following separate job to scrape the cadvisor metrics:

    - job_name: 'kubernetes-cadvisor'

      scheme: https

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}:4194/proxy/metrics

[JorritSalverda]

Having both does lead to duplicate timeline series for kubernetes < 1.7 so make sure to either leave one of them out of your config or include the job in your alerts, rules and graphs. There's interesting info coming from the node itself though, so you probably want to keep scraping those as well from 1.7 onwards.

[unixwitch]

It might make sense to leave the kubernetes-nodes job as is without the cadvisor port and add the following separate job to scrape the cadvisor metrics

This is what the merged PR does, although it doesn't use the cadvisor port since in 1.7.3+ there is a new endpoint for this on the Kubelet metrics port (/metrics/cadvisor). See #2918 for the details.

[brancz]

The separate endpoint on the 4194 port has and will continue to exist though, so I think it's the common denominator in all setups, therefore the current canonical endpoint.

[unixwitch]

I don't think the example configuration should use the cadvisor (port 4191) endpoint. For one thing kubeadm now disables it by default (in 1.7+, I think), so it won't work in any cluster created by kubeadm. We also disable it in our own clusters as we prefer the encrypted and authenticated Kubelet endpoint.

By contrast the Kubelet endpoint (/metrics/cadvisor) is available in all clusters by default.

[brancz]

By contrast the Kubelet endpoint (/metrics/cadvisor) is available in all clusters by default.

This is only true for 1.7.3+ clusters.

But I understand your point. As this is just an example, I'd suggest we move this example to use the /metrics/cadvisor endpoint once it's been established enough (I'd say towards the 1.8 release).

[matthiasr]

My understanding is that 1.7.0…2 are just broken for Prometheus metrics. I don't think we need to support them. We should default to the new endpoint on the authenticated port. It seems most "production grade" cluster distributions disable the unauthenticated port (at least GKE and kubeadm?)

…

On Tue, Aug 1, 2017, 16:20 Frederic Branczyk ***@***.***> wrote: By contrast the Kubelet endpoint (/metrics/cadvisor) is available in all clusters by default. This is only true for 1.7.3+ clusters. But I understand your point. As this is just an example, I'd suggest we move this example to use the /metrics/cadvisor endpoint once it's been established enough (I'd say towards the 1.8 release). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2916 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAICBst_z0-L6Qy_aUXxxNkwT4n4LJEMks5sTzQUgaJpZM4OQyDw> .

[matthiasr]

1.7.3 is out now. the PR for this issue (#2918) is already merged, and the example configuration is very explicit about the different needs for different versions. I think we can close this issue.

[brancz]

Agreed @matthiasr. Thanks everyone!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.