feature request: timeseries blacklist/whitelist for collection

[stapelberg]

When running http://robustirc.net/ in a docker container, the raft package RobustIRC is using (https://github.com/hashicorp/raft) puts the hostname into the metric, and Docker generates hostnames at random. This is a bit unfortunate and I know that for prometheus, the hostname should be a label, but I don’t see a good way to fix this upstream since hashicorp/raft wants to not tie itself to a specific monitoring system.

The downside of this is that everytime I restart my docker containers, I get new timeseries in prometheus, which (for example) makes auto-completion slow and barely usable.

Now, for mitigating this issue, an ideal solution would be to use relabeling to remove the hostname from the metric, but that’s a rather advanced feature.

Instead, since I actually don’t care about the metrics in question, a simpler fix would be to add a blacklist (and/or whitelist) mechanism to prometheus. I.e., for a given target, I’d want to be able to say please ignore all metrics matching /^raftmetrics_[0-9a-f]+_/.

I suspect that situations requiring a filter will become more common as more and more people adopt prometheus in one way or another.

What do you think?

[juliusv]

@stapelberg Yes, adding a blacklist/whitelist filter is definitely planned, and @brian-brazil has advocated for it as well.

Out of curiosity, what are you using to ingest those metrics? Just wondering how the chain of Raft-metric->RobustIRC->(some kind of exporter or native instrumentation?)->Prometheus looks.

[stapelberg]

RobustIRC uses Prometheus directly, so I just configured the /metrics endpoint of each node in my Prometheus instance :).

[juliusv]

@stapelberg Ha! Excellent :) Of course that'd give you the power to rewrite the raft metrics directly when exporting them as Prometheus metrics? But yes, we definitely want filtering support soon. It depends a bit on a larger refactor we're planning to do in the next months to the whole scraping / job configuration / SD layer though.

[brian-brazil]

RobustIRC uses Prometheus directly

I'm a little confused where are the hostnames coming from then? I don't see them in the RobustIRC source.

[stapelberg]

RobustIRC uses Prometheus directly

I'm a little confused where are the hostnames coming from then? I don't see them in the RobustIRC source.

RobustIRC also uses hashicorp/raft, which uses Prometheus via the Prometheus sink in armon/go-metrics — that’s where the hostname comes from.

[juliusv]

@stapelberg Oh, thanks for adding Prometheus to go-metrics! That's awesome.

[brian-brazil]

See #803

[brian-brazil]

http://prometheus.io/docs/operating/configuration/#metric-relabeling-metric_relabel_configs documents this.

[stapelberg]

@brian-brazil So, I’ve tried using this feature; here’s my prometheus configuration:

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 10s

rule_files:
- robustirc_prometheus.rules

scrape_configs:

- job_name: prometheus
  scheme: http
  target_groups:
  - targets:
    - 'localhost:9090'

- job_name: robustirc
  scheme: https
  basic_auth:
    username: <hidden>
    password: <hidden>
  dns_sd_configs:
  - names:
    - '_robustirc._tcp.robustirc.net'
  relabel_configs:
  - source_labels: ['__address__']
    regex:         '(.+)\.robustirc\.net:[0-9]+'
    target_label:  'instance'
    replacement:   '$1'
  metric_relabel_configs:
  - regex:  '^raftmetrics_[0-9a-f]+_'
    action: drop

However, I still see new raftmetrics_ timeseries cropping up, most recently raftmetrics_1ef6b3f90088_runtime_alloc_bytes.

What am I doing wrong here? Shall I open a new issue for this problem?

[brian-brazil]

This is a consequence of us anchoring all the relabelling and selection regular expressions. Add a .* to the end of the regex.

[stapelberg]

Ah. Does that also mean the ^ in the beginning should be removed?

Also, can we update the docs to reflect that?

[jimmidyson]

Would it make sense for the anchoring to check whether the provided regex is already anchored?

[brian-brazil]

This was documented in prometheus/docs#213

Extra anchoring is harmless, and trying to do magic and guess what the user wants is likely to cause more problems than it solves.

[jimmidyson]

Fair comment.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.