selective metric drop with relabel config

[lastsky]

node_systemd_unit_state is expensive_metric so I spent about 4h for googling and testing different configs and trying to create rule:

keep:

node_systemd_unit_state{name="nginx.service",state="active"} 1
node_systemd_unit_state{name="sshd.service",state="active"} 1

drop all others:

<...>
node_systemd_unit_state{name="nginx.service",state="deactivating"} 0
node_systemd_unit_state{name="nginx.service",state="failed"} 0
node_systemd_unit_state{name="nginx.service",state="inactive"} 0
node_systemd_unit_state{name="systemd-logind.service",state="active"} 1
node_systemd_unit_state{name="systemd-logind.service",state="deactivating"} 0
node_systemd_unit_state{name="systemd-logind.service",state="failed"} 0
node_systemd_unit_state{name="systemd-logind.service",state="inactive"} 0
<...>

• Prometheus version:

version 1.7.1

• Prometheus configuration file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 15s
    file_sd_configs:
        - files:
            - '/etc/prometheus/file_sd_configs/*.yml'

rule_files:
    - '/etc/prometheus/rules/*.conf'

not working strings:

- metric_relabel_config:
    - source_labels: [__name__, name, state]
      regex: node_systemd_unit_state;nginx.service;active
      relabel_action: keep

    - source_labels: [__name__, name, state]
      regex: node_systemd_unit_state;sshd.service;active
      relabel_action: keep

   - source_labels: [__name__]
     regex: node_systemd_unit_state
     relabel_action: drop

target sources for selective label drop:

# cat /etc/prometheus/file_sd_configs/hardware.yml 
- targets:
    - "10.137.161.82:9100"
    - "10.137.161.83:9100"
    - "10.137.161.84:9100"
    - "10.137.161.85:9100"
  labels:
    job: node

I'd tested about ~20 examples, and after time curl -XDELETE -g 'http://localhost:9090/api/v1/series?match[]=node_systemd_unit_state' I see them in prometheus again. If prometheus starts, of course.

Would you like to explain where I should insert metric_relabel_config? to prometheus.yml or to static configs? Please add working example to docs.

Tired. Thanks.

[lastsky]

another 4 hours of googling examples.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 15s
    file_sd_configs:
        - files:
            - '/etc/prometheus/file_sd_configs/*.yml'

    metric_relabel_configs:
      - source_labels: [__name__,name]
        regex:         node_systemd_unit_state;nginx.*
        action:        drop

this works for drop labels like:

node_systemd_unit_state{name="nginx.service",state="activating"} 0
node_systemd_unit_state{name="nginx.service",state="active"} 1
node_systemd_unit_state{name="nginx.service",state="deactivating"} 0
node_systemd_unit_state{name="nginx.service",state="failed"} 0
node_systemd_unit_state{name="nginx.service",state="inactive"} 0

ok. but any types of exclude inside regex doesnt work.

[lastsky]

so temporary solution is -collector.systemd.unit-whitelist=(nginx.service|ssh.service) but I would like to know the correct solution for flexible and selective metric drops in scrape_configs.

how to:

1. keep node_systemd_unit_state for list of services and state="active":

node_systemd_unit_state{name="nginx.service",state="active"} 1
node_systemd_unit_state{name="sshd.service",state="active"} 1

2. drop other node_systemd_unit_state*:

<...>
node_systemd_unit_state{name="nginx.service",state="deactivating"} 0
node_systemd_unit_state{name="nginx.service",state="failed"} 0
node_systemd_unit_state{name="nginx.service",state="inactive"} 0
node_systemd_unit_state{name="systemd-logind.service",state="active"} 1
node_systemd_unit_state{name="systemd-logind.service",state="deactivating"} 0
node_systemd_unit_state{name="systemd-logind.service",state="failed"} 0
node_systemd_unit_state{name="systemd-logind.service",state="inactive"} 0
<...>

3. and keep all other metrics from target: node_.*, process_.* etc.

read node_systemd_unit_state as another_i_fucked_this_target_expensive_metric if you wish.

[lastsky]

this is NOT urgent. but please provide an example. thanks.

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.