Multiple relabel configs with action: keep results in all metrics being dropped

[roganartu]

Problem

The following set of relabel rules results in all metrics being dropped:

metric_relabel_configs:
      - source_labels: ['__name__']
        regex: foo_total
        action: keep
      - source_labels: ['__name__']
        regex: bar_total
        action: keep

According to the documentation, this behaviour is entirely expected and technically correct, but I don't think it's very user-friendly.

The relabel function only looks at a single config at a time, so the first rule drops everything that matches the second, and the second drops everything that matches the first: https://github.com/prometheus/prometheus/blob/master/relabel/relabel.go#L40

Workaround

metric_relabel_configs:
      - source_labels: ['__name__']
        regex: '(foo_total|bar_total)'
        action: keep

However this obviously quickly becomes cumbersome with any kind of complexity in regex and/or source labels, eg: for a (likely to grow and change over time) whitelist of metric-label combinations in remote_write.

Proposal

Make action: keep behaviour more consistent with it being the conceptual inverse of action: drop by keeping all metrics that match any action: keep config, not just those that match both. I think that action: keep configs OR'd together are likely more useful than the existing AND behaviour, but I have no data to back that up.

This breaks existing behaviour in two obvious ways, and perhaps more subtle ways:

1. The existing AND-style behaviour (eg: chaining progressively less restrictive action: keep configs together instead of a single long one will no longer work)
2. A fix likely requires processing all the action: keep configs together either before or after other relabel rules are applied. This will break configs that rely on order to work correctly (eg: using an initial action: keep followed by other configs that assume metrics to have been dropped).

[brian-brazil]

Do you have a proposal?

[roganartu]

edited to add proposal

[brian-brazil]

That's a breaking change, which we can't do and would cause confusion. It's also a complete change in semantics for how actions work, as they're currently independent. If you want multiple keeps ORed together, you can already combine them into one action (and it's probably more efficient too).

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.