etcd serves stale read requests before log is caught up during recovery/startup

[spacejam]

Steps to reproduce:

1. write many monotonically increasing values to a key
2. restart server while trying to read the key in a loop
3. observe several values returned as the log catches up

It should wait until the log has caught up before serving reads.

[xiang90]

@spacejam If you want to prevent stale read, you need to specify quorum=true.

[philips]

I think people are looking for an SLA for how "fresh" the results are. For example if we can give someone a GET request that only succeeds if the last successful heartbeat RPC was X seconds ago.

[spacejam]

Is there a case where it is desirable to serve read requests before the local state is fully recovered? This is likely to involve serving state that is far older than typical follower lag.

[xiang90]

@spacejam I am curious about the use case. If user really care about the "freshness" more than linearizability, it might make sense to add a feature to block on the client request until:

1. fully applied existing entires
2. or fully caught up the current leader
3. or caught up with current leader at least N second ago

Or user can simply specify quorum=true.

But I am not convinced that user needs this fine grained control.

[spacejam]

I think people are generally fine with some slave lag, and would prefer stale data to no data in the presence of partitions, but I don't think many people are willing to trade extremely stale reads for a few moments of acausal availability in the rare case of server start-up. This is why many databases will not serve requests until they have fully recovered their local state upon initialization.

[xiang90]

@spacejam Oh. Yea. For startup initialization, it is an easy fix and should be fixed.

It should wait until the log has caught up before serving reads.

Your original issue is border than initialization if I understand it correctly.

A server can recover from snapshot at runtime too. Shall we wait until it catches up with the leader?

A server might be partitioned and is far behind the leader. Shall we wait again?

I have some rough answers in my mind. But I do not have to plan to address them immediately.

[xiang90]

From #3300:

etcd serves stale read requests before log is caught up during startup.

[xiang90]

Closed by #5196