Useful labels missing from Prometheus node_exporter in 1.7.x

[ericuldall]

What did you do?
Upgraded to GKE 1.7.*

What did you expect to see?
Same metrics as before

What did you see instead? Under which circumstances?
Missing lots of meta data (labels) from my metrics

• Prometheus configuration file:

prometheus.yml: |-
    global:
      scrape_interval: 15s
    scrape_configs:
    - job_name: 'kubernetes-nodes'

      scheme: https

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics
    - job_name: 'kubernetes-cadvisor'
      scheme: https

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}:4194/proxy/metrics
    - job_name: 'kubernetes-apiservers'
      kubernetes_sd_configs:
      - role: endpoints
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)(?::\d+);(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_service_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name
    - job_name: 'kubernetes-services'
      scheme: https
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_service_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: kubernetes_name
    - job_name: 'kubernetes-pods'
      scheme: https
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (.+):(?:\d+);(\d+)
        replacement: ${1}:${2}
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_pod_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

In my previous version of GKE < 1.7 I'm able to see lots of labels associated with node_exporter info. Namely node_cpu at this point.

Here's an example of an older version:

node_cpu{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="n1-standard-2",beta_kubernetes_io_os="linux",cloud_google_com_gke_nodepool="default-pool",cpu="cpu1",failure_domain_beta_kubernetes_io_region="europe-west1",failure_domain_beta_kubernetes_io_zone="europe-west1-b",instance="my-cluster-default-pool-3cbb3136-22jh",job="kubernetes-node-exporter",kubernetes_io_hostname="my-cluster-default-pool-3cbb3136-22jh",mode="system"}

As you can see there's tons of useful information there I can group by. Most important to me is the instance name, node pool and region/zone.

Here's an example from a 1.7 cluster:

node_cpu{app="node-exporter",cpu="cpu1",instance="10.128.0.25:9100",job="kubernetes-service-endpoints",kubernetes_name="node-exporter",mode="system",name="node-exporter"}

In the newer version I'm missing all of the context about the node. The best I can do with this data is to group on instance and in my legend I'll only see a list of internal ip's instead of hostnames.

I have filed this issue previous on the node-exporter issues previously: prometheus/node_exporter#678 At that pointed they requested I open an issue with GKE: kubernetes/kubernetes#52858 and finally Kubernetes team is requesting I open this ticket...so I'm a bit lost in the mix here.

Any help is much appreciated.

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[sl1pm4t]

poor @ericuldall has been ping ponged all over the internet seeking an answer to this one

[brian-brazil]

This is mostly likey a Prometheus usage question, and usage questions are handled on the mailing list. You must appreciate that we get several usage questions a day incorrectly filed as bugs, so we can't divert too much attention from development work for all of them.

[ericuldall]

No complaints here. Just looking for someone who knows the answer 👍
I've been a bit distracted from this hunt lately. I'll head over to the mailing list soon!

[ericuldall]

I've posted my question in the group as suggested: https://groups.google.com/forum/#!topic/prometheus-users/2njYh6SucJ8

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.