[kube-prometheus-stack] add attach_metadata to kubelet related scrape targets

[youwalther65]

Is your feature request related to a problem ?

Prometheus configuration kubernetes_sd_config allows attaching optional metadata to discovered targets , see here

# Optional metadata to attach to discovered targets. If omitted, no additional metadata is attached.
attach_metadata:
# Attaches node metadata to discovered targets. Valid for roles: pod, endpoints, endpointslice.
# When set to true, Prometheus must have permissions to get Nodes.
  [ node: <boolean>) | default = false ]

I 'd like to use this for kubelet related scrape targets as well.

Describe the solution you'd like.

In Helm values.yaml like:

kubelet:
  enabled: true
  namespace: kube-system
  serviceMonitor:
    attachMetadata:
      node: true

Describe alternatives you've considered.

I disabled kubelet and created separate "additionalScrapeConfigs" like:

prometheus:
  # high avialability setup
  prometheusSpec:
    additionalScrapeConfigs:
    - job_name: k8s-kubelet
      kubernetes_sd_configs:
        - role: node
      honor_labels: true
      scheme: https
      metrics_path: /metrics
      tls_config:
        insecure_skip_verify: false
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels:
        - __meta_kubernetes_node_name
        action: replace
        target_label: node
        regex: (.*)
        replacement: ${1}
      - source_labels:
        - __metrics_path__
        action: replace
        target_label: metrics_path
    - job_name: k8s-kubelet-cadvisor
      kubernetes_sd_configs:
        - role: node
      honor_labels: true
      scheme: https
      metrics_path: /metrics/cadvisor
      tls_config:
        insecure_skip_verify: false
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels:
        - __meta_kubernetes_node_name
        action: replace
        target_label: node
        regex: (.*)
        replacement: ${1}
      - source_labels:
        - __metrics_path__
        action: replace
        target_label:  metrics_path
      metric_relabel_configs:
      - action: drop
        regex: container_cpu_(load_average_10s|system_seconds_total|user_seconds_total)
        source_labels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        source_labels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        source_labels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        source_labels:
        - __name__
      - action: drop
        regex: container_spec.*
        source_labels:
        - __name__
      - action: drop
        regex: .+;
        source_labels:
        - id
        - pod
    - job_name: k8s-kubelet-probes
      kubernetes_sd_configs:
        - role: node
      honor_labels: true
      scheme: https
      metrics_path: /metrics/probes
      tls_config:
        insecure_skip_verify: false
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels:
        - __meta_kubernetes_node_name
        action: replace
        target_label: node
        regex: (.*)
        replacement: ${1}
      - source_labels:
        - __metrics_path__
        action: replace
        target_label:  metrics_path
...
## Component scraping the kubelet and kubelet-hosted cAdvisor
##
kubelet:
  enabled: false
...

Additional context.

I want to filter kubelet and cAdvisor metrics based on additional labels like node group name etc.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[jcpunk]

I'd still like to see this resolved.

[zeritti]

Prometheus currently does not support attaching metadata to target endpoints that do not set nodeName which is the case of the endpoints created by Prometheus operator (targetRef.kind=Node gets set at these endpoints). PR #12338 addressing this has recently been merged. Once released, the kubelet's service monitor will be adjusted accordingly.

[QuentinBisson]

This has been released in prometheus 2.44 which is now in the latest chart version. @jcpunk can you check?

[youwalther65]

Thank you all. At least what I have proposed is not yet visible in latest kube-prometheus-stack Helm values.yaml

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[DmytroKozlovskyi]

Hi, For us it looks like a bug and not an enhancement. We are trying to migrate to Prometheus operator from the regular Prometheus helm chart because of missing backward compatibility. Without this parameter, we can't return back all required node labels which are used in many places in our environment (alerting rules, Grafana dashboards, etc).