Expected behaviour on delayed replication

[balbirthomas]

Hi,

summary

I am trying to understand what is the expected behavior when an additional Mimir instance is added to the memberlist at a later point in time. I did an experiment (detailed below) to understand this behavior. I would be grateful to know if the behavior I observe is as expected.

experiment

• I initially deployed a single Mimir monolithic instance using. Configuration for this is attached here. I then used grafana agent to push metrics to this instance for some time, say 15 minutes.
• Now I deployed a second Mimir monolithic instance and also updated the configuration of the first instance, to ensure that the member list of both instances had 2 memberlist members and the replication factor was 2. The Mimir configuration file for this is
  attached here.
• Now I used Grafan to query each of the two instances.

observations

Here are my observations

• If I query the first instance (for example: an instant vector of some counter like process_cpu_seconds) I see all metrics including those for the first 15 minutes during which only the first Mimir instance was running .
• If I query the second instance (using the same query) I only see metrics since this second instance was started. The query result is missing metrics data for the first 15 minutes.

questions

1. Is this expected behavior ?
2. My understanding was if Grafana sends a query to any Mimir instance, the query is forwarded to all ingestors in the memberlist in addition to object storage. Hence I expected both instances should return the same query result. What am I missing ?
3. Is there any way to ensure that queries sent to the second Mimir instance return the same results as the first instance (effectively making the second instance a full replica of the first) ?

[pstibrany]

The problem with replication factor 2 is that on write, it requires both replicas to successfully receive samples, so that we can tolerate single failure on read. However in your scenario, only one replica received samples. During read as soon as data from single replica is received, read finishes. Depending on which replica replies, you either get results or not.

Re 2: you are right that query goes to all ingesters, but as soon as single one replies (with RF=2), this reply is sent back as final answer, even if reply is empty.

Re 3: Mimir currently doesn't have any way of "syncing" replicas.

[pracucci]

The problem with replication factor 2 is that on write

Keep in mind that replication factor = 2 has never been seriously tested in Mimir. I'm personally not 100% sure it even works as expected. I'm wondering if we should just add a validation check and forbid using it at all in Mimir.

[pstibrany]

I'm personally not 100% sure it even works as expected.

I don't think it does. It currently allows 1 failed replica on both write and read, which can result in inconsistencies.