Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persistent subscriptions: Ignore replayed events during checkpointing #1890

Merged
merged 6 commits into from Apr 30, 2019

Conversation

2 participants
@shaan1337
Copy link
Member

commented Mar 11, 2019

Fixes #500

This PR fixes a bug occurring in persistent subscription where an invalid checkpoint is written when parked events are replayed and the size of the parked stream is larger than the size of the original stream.

Replaying parked messages multiple times will make the issue more likely to happen (assuming the parked stream is not empty).

Symptoms

  • In the Persistent subscription dashboard, Negative values appear for # of msgs and Current is greater than Known

Root cause
Event number of parked stream is being checkpointed by the persistent subscription

Impact

  • Persistent subscription checkpoint is written with a value ahead of last stream's event number (The checkpoint is actually set to the parked stream event number instead of the original stream's event number)
  • Restarting a node after a wrong checkpoint may cause events to be missed by the subscription.

Reproduction steps

  1. Create a persistent subscription with settings:
  • Stream: x
  • Group: group
  • Message Timeout: 1 ms
  • Checkpoint after: 1 ms
  • Min checkpoint count: 1
  • Max checkpoint count: 1
  1. Connect a client with autoAck: true.

  2. Write 10 events to stream x. The client should receive all events.

  3. Verify that checkpoint is set to 9 in stream: $persistentsubscription-x::group-checkpoint

  4. Stop the client, set autoAck: false and start the client again.

  5. Write 10 events to stream x.
    All events should be parked in $persistentsubscription-x::group-parked and checkpoint should be set to 19 in $persistentsubscription-x::group-checkpoint

Now comes the important part:
7. Replay all parked events. This should not affect the checkpoint stream since we're only replaying events that have already been processed. As long as the size of the parked stream $persistentsubscription-x::group-parked is smaller than the original stream x, the checkpoint will not be written.

Replay parked events a few more times and you should see a checkpoint being written with the event number from the parked stream in $persistentsubscription-x::group-checkpoint. The UI will also start showing Current > Known and negative values for # of msgs.

Resolution
Ignore replayed/parked events when checkpointing by adding an IsReplayedEvent property to OutstandingMessage class

@shaan1337 shaan1337 requested review from jen20 and jageall Apr 30, 2019

@jageall jageall merged commit 9e1311f into master Apr 30, 2019

9 checks passed

EventStore.EventStore Build #20190311.1 succeeded
Details
EventStore.EventStore (Centos 7 x64 Debug) Centos 7 x64 Debug succeeded
Details
EventStore.EventStore (Centos 7 x64 Release) Centos 7 x64 Release succeeded
Details
EventStore.EventStore (Ubuntu 14.04 x64 Debug) Ubuntu 14.04 x64 Debug succeeded
Details
EventStore.EventStore (Ubuntu 14.04 x64 Release) Ubuntu 14.04 x64 Release succeeded
Details
EventStore.EventStore (Windows x64 Debug) Windows x64 Debug succeeded
Details
EventStore.EventStore (Windows x64 Release) Windows x64 Release succeeded
Details
EventStore.EventStore (macOS x64 Debug) macOS x64 Debug succeeded
Details
EventStore.EventStore (macOS x64 Release) macOS x64 Release succeeded
Details

@shaan1337 shaan1337 deleted the persistentsub-parked-fix branch May 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.