Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix idempotent writes for cases where events have not yet been replicated #1915

Merged
merged 2 commits into from
Apr 30, 2019

Conversation

shaan1337
Copy link
Member

@shaan1337 shaan1337 commented Apr 24, 2019

The following scenario can occur in a cluster (let's assume a 3 node cluster in this example):

  • Client tries to write an event A
  • The master receives event A, write it locally which is replicated and waits for commit acks from slaves
  • Both slaves go down at approximately same time and a quorum number of commit acks are never received by the master (in a 3 node cluster, one commit ack from a slave would be enough)
  • Client sees a timeout and tries to write event A again
  • The master sees it as an idempotent write and immediately replies success although the event has not yet been replicated.
  • If we elaborate the scenario further more and assume the master then goes down and the two slaves come up and form a cluster, this event will be lost since the original master will truncate when rejoining the cluster.

Proposed fix (in commit a0533bc)

  • Before returning CommitDecision.Idempotent, verify if the event has been committed to the index (which is an indicator of whether the write has been replicated: as from v4.1.0 events are committed to the index on the master only when enough commit acks have been received to prevent dirty reads from the master)
  • Otherwise, return a new decision: CommitDecision.IdempotentNotReady
  • Handle CommitDecision.IdempotentNotReady by just dropping the write and wait for the client to try again.

Tests
Several tests have also been added around idempotency.

jageall
jageall previously approved these changes Apr 25, 2019
@jageall jageall merged commit 16caa75 into master Apr 30, 2019
@shaan1337 shaan1337 deleted the fix-idempotent-writes branch May 13, 2019 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants