Detect/forbid (verbatim) duplicated lines in the rules files

[beorn7]

Accidentally, I found a case where the same rule name was used twice. That went without complaints by the server, and if both rule evaluations had resulted in different label sets, it might have just worked. However, they resulted in the same label set, so the results were funneled into the same time series, and now with our fancy monotonicity detector, we actually see warnings.

Question: Should we in general forbid to use the same rule name twice? Or are their use cases where you want to use the same rule name (but create different label values for the result)?

I tend to simply ban duplicate rule names, but perhaps I'm missing something.

@juliusv @brian-brazil

[brian-brazil]

I'd say the current behaviour is correct.

[beorn7]

Found more duplicate rule files at SC... happens quite often, and goes completely undetected so far.

@brian-brazil What's the use case for the current behavior?

[brian-brazil]

As you said in the OP, when you have different label values for each.

[beorn7]

Let me rephrase: What would be a use case for creating two rules with the same name that have different label values?

[brian-brazil]

If different teams had per-job rules, those applying to the Standard Exports would be in this situation.

[beorn7]

I guess that's a valid point... we have one server per team at SC, so basically one team manages all the rules on one server.

But we could detect if a rule is completely identical to another one. Would that make sense?

[brian-brazil]

We could, but it'd break the abstraction that each rule runs every eval interval. It'll also cause oddities when we have ordered rule evaluation.

[beorn7]

I mean, we could simply reject to load the rules file if we have an exact duplicate.

[brian-brazil]

I'm not sure there's much value to that - should we also bail out if there's two rules that are effectively equal too?

[beorn7]

It might be difficult to detect “effectively equal” rules.

But on at least three servers at SC, I found rule pairs that funneled into the same time series. One of them was basically a typo, i.e. two different expressions put their results into the same time series (bad). The other two were exact duplicates (copy&paste problem). The related time series had therefore duplicated samples, and with 0.15.0, those samples got rejected, and a counter got incremented on which you probably want to alert. So catching exactly duplicated rules will help to prevent a mistake that will later page you...

[matthiasr]

I did use this feature (no quotes) to feed different queries / metrics into one uber-metric with different labels (@beorn7 this is on a branch r/n, I can show you). I think we should keep allowing them, and we can't tell beforehand that the labelsets will be different.

I would vote to make a note of this in the documentation but allow people shooting themselves in the foot.

[matthiasr]

… and if people get paged for this then that's exactly the right point in time to take note of it, because it's the only point when we can be sure it is a problem.

[juliusv]

Agreed, this sounds more like an example of bad external rule / config management than one with Prometheus per se. Would also vote to keep the current behavior.

[beorn7]

Still, if i have two or more literally (in the literal sense of "literally") identical rules, there is no way that can be meaningful. So should we at least error in that case?

[juliusv]

I'd be ok with erroring in that case. I'd say the usefulness is limited (slight decrease in computing resources, but the storage will just reject the second copy of identical samples anyways now), but it couldn't hurt.

[beorn7]

I have adjusted the subject.

[grobie]

👍 to being strict about such issues

[brian-brazil]

This is probably something for a warning in promtool, once we add a general rule linter

[gouthamve]

How does this play with rule-groups?

Duplicated rule in rule-group would be an error as all rules in the rule-group are evaluated with the same timestamp. Also, duplicated rule across groups can also cause out-of-order appends.

[brian-brazil]

Duplicated rule in rule-group would be an error as all rules in the rule-group are evaluated with the same timestamp. Also, duplicated rule across groups can also cause out-of-order appends.

Not necessarily, each rule could be one iteration of a state machine and you can construct those to avoid duplicate timestamps.

[gouthamve]

But how would avoid the entire rule-group being evaluated at the same timestamp? The same rule in different groups might make sense, but the same rule in the same group will be either a nop or an error right?

[brian-brazil]

You're thinking of normal simple rules, self-referential rules could have different effects when repeated.

[totallyunknown]

Can we add a duplicate rule name detection into promtool?