Prometheus server high CPU usage for Label sorting

[mxork]

Using prometheus as monitor on kubernetes cluster. ~100 node_exporters, ~300 targets total. Running go tool pprof --gif $server/debug/pprof/profile > prof.gif shows high CPU usage for sorting Labels.

We're running prometheus as a fairly by-the-books kubernetes custom-metrics-apiserver, using k8s-prometheus-adapter.

The code (https://github.com/prometheus/prometheus/blob/master/pkg/labels/labels.go) seems to have two reasons for running slowly: one is passing through Sort interface, the other is the repeated comparison of strings with a possibly shared prefix. Using labels to represent paths in a hierarchy seems like a reasonable use case, and shouldn't lead to poor performance. If anyone has a one-liner to generate the list of labels being operated on, I can dump them.

If my diagnosis is correct, there are low-hanging solutions. One is to go:generate a sort procedure which does not pass through the interface barrier (several generators are floating around). This is simplest, but does not address the shared-prefix problem. Two is to implement a trie-sort, which does.

System information:

Centos 7.4
Linux 3.10.0-693.11.6.el7.x86_64 x86_64

Prometheus version:

2.1.0

Prometheus configuration file:

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
rule_files:
- /etc/config/rules
- /etc/config/alerts
scrape_configs:
- job_name: prometheus
  static_configs:
  - targets:
    - localhost:9090
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  job_name: kubernetes-apiservers
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
    regex: default;kubernetes;https
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_service_name
    - __meta_kubernetes_endpoint_port_name
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  job_name: kubernetes-nodes
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - replacement: kubernetes.default.svc:443
    target_label: __address__
  - regex: (.+)
    replacement: /api/v1/nodes/${1}/proxy/metrics
    source_labels:
    - __meta_kubernetes_node_name
    target_label: __metrics_path__
  scheme: https
  tls_config:

[brian-brazil]

Can you try with a newer version of Prometheus? You're probably hitting an issue with SD that was fixed.

[mxork]

Just bumped to 2.2.0. Node doesn't appear at all.

Thanks for the quick response! Glad sorting wasn't actually a bottleneck.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.