KV Read-after-Write Consistency #6557
Description
Proposed change
Since introducing Direct Get we lost our, admittedly imperfect, read-after-write consistency in KV stores.
We felt then that was an acceptable behaviour change but its clear that there are significant need for control in this area.
We might still default to the current behaviour but should at least provide a opt-in to be in-cluster consistent and in-supercluster eventually consistent.
This can take a few forms:
- We can add a leader-only direct get subject
- We can adopt some of the thoughts @MauriceVanVeen has around improving the consistency of the raft layer
Even if we do nothing, we should very clearly and prominently document what we do and do not support in places other than the ADR.
Use case
Increase the areas KV is applicable
Contribution
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
Participants
Activity
MauriceVanVeen commentedon Feb 21, 2025
If we'd be able to only get a response from a leader, we'd still not be able to guarantee read-after-write.
Without leader changes it'd work, but during every leader change there's a chance you get stale reads. Also if the current leader is network partitioned, and you write using a different connection to the new leader, you'd also get stale reads from the old (partitioned) leader.
There are a number of options to be able to guarantee this, even without requiring you to always connect directly to the leader. @mprimi has some thoughts about that as well, allowing replicas to answer if they can guarantee your read-after-write (with of course other tradeoffs as well).
Using the same connection object to guarantee read-after-writes sounds okay, but likely turns out to be brittle. Instead we should probably go for an implementation where we can strictly guarantee this. Regardless of if only the leader answers, or replicas can also answer.
De-flake TestNoRaceJetStreamKVReplaceWithServerRestart (#6947)
MauriceVanVeen commentedon Jun 25, 2025
While adding support in the Go client.. I realised we must also support read-after-write for consumers!
For example, KV PUT a value, and then do KV Keys. You expect to get back the key you've just put. But we can't currently guarantee that either, the consumer will eventually get that data but because Keys stops when it reaches
pending=0it quits too early and doesn't contain the written value.The solution should solve read-after-write and monotonic reads for direct/msg get requests and read-after-write for consumers.
(2.12) Read-after-write & monotonic reads (#6970)
(2.12) Remove Read-after-Write for now (#7167)