Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix idempotent writes for cases where events have not yet been replicated #1915

Merged
merged 2 commits into from Apr 30, 2019

Conversation

2 participants
@shaan1337
Copy link
Member

commented Apr 24, 2019

The following scenario can occur in a cluster (let's assume a 3 node cluster in this example):

  • Client tries to write an event A
  • The master receives event A, write it locally which is replicated and waits for commit acks from slaves
  • Both slaves go down at approximately same time and a quorum number of commit acks are never received by the master (in a 3 node cluster, one commit ack from a slave would be enough)
  • Client sees a timeout and tries to write event A again
  • The master sees it as an idempotent write and immediately replies success although the event has not yet been replicated.
  • If we elaborate the scenario further more and assume the master then goes down and the two slaves come up and form a cluster, this event will be lost since the original master will truncate when rejoining the cluster.

Proposed fix (in commit a0533bc)

  • Before returning CommitDecision.Idempotent, verify if the event has been committed to the index (which is an indicator of whether the write has been replicated: as from v4.1.0 events are committed to the index on the master only when enough commit acks have been received to prevent dirty reads from the master)
  • Otherwise, return a new decision: CommitDecision.IdempotentNotReady
  • Handle CommitDecision.IdempotentNotReady by just dropping the write and wait for the client to try again.

Tests
Several tests have also been added around idempotency.

@shaan1337 shaan1337 requested review from jen20, daveremy and jageall Apr 24, 2019

@shaan1337 shaan1337 force-pushed the fix-idempotent-writes branch from 6217284 to f7f01bb Apr 30, 2019

@shaan1337 shaan1337 requested a review from jageall Apr 30, 2019

@jageall jageall merged commit 16caa75 into master Apr 30, 2019

9 checks passed

EventStore.EventStore Build #20190430.1 succeeded
Details
EventStore.EventStore (Centos 7 x64 Debug) Centos 7 x64 Debug succeeded
Details
EventStore.EventStore (Centos 7 x64 Release) Centos 7 x64 Release succeeded
Details
EventStore.EventStore (Ubuntu 14.04 x64 Debug) Ubuntu 14.04 x64 Debug succeeded
Details
EventStore.EventStore (Ubuntu 14.04 x64 Release) Ubuntu 14.04 x64 Release succeeded
Details
EventStore.EventStore (Windows x64 Debug) Windows x64 Debug succeeded
Details
EventStore.EventStore (Windows x64 Release) Windows x64 Release succeeded
Details
EventStore.EventStore (macOS x64 Debug) macOS x64 Debug succeeded
Details
EventStore.EventStore (macOS x64 Release) macOS x64 Release succeeded
Details

@shaan1337 shaan1337 deleted the fix-idempotent-writes branch May 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.