Adding redundancy with de-duplication to Mimir Ruler?

[rudo-thomas]

What We're Trying to Achieve

We have set up a Mimir v2.1.0 stack including a Ruler in remote mode. Our
undestanding is that in steady state a particular rule group is evaluated by
the Ruler pool exactly once. Briefly, evaluation cycles might be missed during
Ruler replica restarts, resulting in "holes" in the data.

Our goal is to eliminate those holes. To achieve that, we'd like to add
redundancy: Evaluate each rule group twice, then de-duplicate the results.

Question

What would be the recommended approach to implementing the redundancy as
described above?

Ideas

We've come up with the following ideas on how to implement the redundancy.

HA Tracker

TL;DR have two pools of Rulers with the same number of workers; the pair of
corresponding workers (one from each pool) would choose a leader using HA
tracker -- achieving the de-duplication.

For such de-duplication to work well, a particular rule group would always have
to hash to the same replica in both pools. In other words, each pair of workers
competing for leadership has to be evaluating the same set of rule groups.

This approach would require changes in how the ring is used by the Ruler pools
(e.g. when a worker is down, other workers in the same pool must not pick up
its work). Also, resizing the pools -- without introducing holes and/or
duplicate data -- would be a challenge.

Compactor

TL;DR have two pools of Rulers, have each pool (as a whole) write to a disjoint
set of Ingesters (maybe separate pools?), then let the Compactor de-duplicate.

---

Thanks for any pointers.

[pracucci]

HA Tracker

The HA tracker is implemented through a leader election. A new leader is elected after no data is received from the current leader for X time (configurable timeout). Due to the nature of ruler (rule evaluations are regular intervals) I think you can't set a low timeout, and so you could end up with missing rule samples anyway. Happy to be proved to be wrong.

Compactor
TL;DR have two pools of Rulers, have each pool (as a whole) write to a disjoint set of Ingesters (maybe separate pools?), then let the Compactor de-duplicate.

Compactor (as it works today) will not de-duplicate samples, because sample timestamps for series generated by both ruler replicas will be different.

[pstibrany]

What do you say if we introduce per-tenant limit for number of rulers per rule group, and we test it internally for our tenant? One negative result is that it may require more queriers/query-frontends and double the effort on ingesters.

[pstibrany]

We should also avoid rolling out multiple rulers in that case, to avoid one of them being down.

[pracucci]

What do you say if we introduce per-tenant limit for number of rulers per rule group, and we test it internally for our tenant?

Speaking about Grafana Labs priorities we can further discuss it internally. However, this idea can also be experimented by OSS community too.

That being said, I'm wondering what happens to alerting rules, being double notified to Alertmanager (should be deduplicated, and nothing bad should happen, but it's something to double check).

One negative result is that it may require more queriers/query-frontends and double the effort on ingesters.

That's for sure. It will double the cost of evaluating rules.

We should also avoid rolling out multiple rulers in that case, to avoid one of them being down.

We already do:

mimir/operations/mimir/ruler.libsonnet

Lines 49 to 50 in 7e61f6d

	deployment.mixin.spec.strategy.rollingUpdate.withMaxSurge(0) +
	deployment.mixin.spec.strategy.rollingUpdate.withMaxUnavailable(1) +

[codesome]

Rule evaluation timestamps are not actually aligned with the interval, they are just equally spaced.
See this: https://github.com/prometheus/prometheus/blob/d41e5a558294666ba572648b454b5f7a5f548573/rules/manager.go#L534-L542
There is a jitter based on the rule group hash

[codesome]

As long as the rule groups as exactly same (i.e. their .hash() remains same), then the jitter should also be consistent across replicas.