Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

!!! BUGFIX: Release "highest applied sequence number" during catchup #262

Conversation

@bwaidelich
Copy link
Member

bwaidelich commented Mar 20, 2020

Previously the whole EventListenerInvoker::catchup() call triggered a
single blocking transaction that was only committed when the corresponding
event listener was up to date with the event store.
Especially for replays and initial catchups this posed a problem because
of the ever growing transaction.

With this change the "Highest Applied Sequence Number" is released after
each applied event.
This has the nice side effect that multiple workers can share the load:
While process 1 commits the transaction, process 2 can already start processing
the next event.

For batch-processing the batch size can be increased in order to improve performance:

$eventListenerInvoker = (new EventListenerInvoker($eventStore, $listener, $connection))
   ->withTransactionBatchSize(500)
   ->catchup()

This is a breaking change because the signature of the EventListenerInvoker has
changed slightly:

Previously:

$eventListenerInvoker = new EventListenerInvoker($eventStore);
$eventListenerInvoker->catchup($listener, $progressCallback);

Now:

$eventListenerInvoker = new EventListenerInvoker($eventStore, $listener, $connection);
$eventListenerInvoker->onProgress($progressCallback);
$eventListenerInvoker->catchup();

Fixes: #261

bwaidelich added 2 commits Mar 20, 2020
Previously the whole `EventListenerInvoker::catchup()` call triggered a
single blocking transaction that was only committed when the corresponding
event listener was up to date with the event store.
Especially for replays and initial catchups this posed a problem because
of the ever growing transaction.

With this change the "Highest Applied Sequence Number" is released after
each applied event.
This has the nice side effect that multiple workers can share the load:
While process 1 commits the transaction, process 2 can already start processing
the next event.

For batch-processing the batch size can be increased in order to improve performance:

    $eventListenerInvoker = (new EventListenerInvoker($eventStore, $listener, $connection))
       ->withTransactionBatchSize(500)
       ->catchup()

This is a breaking change because the signature of the `EventListenerInvoker` has
changed slightly:

Previously:

    $eventListenerInvoker = new EventListenerInvoker($eventStore);
    $eventListenerInvoker->catchup($listener, $progressCallback);

Now:

    $eventListenerInvoker = new EventListenerInvoker($eventStore, $listener, $connection);
    $eventListenerInvoker->onProgress($eventListenerInvoker);
    $eventListenerInvoker->catchup();

Fixes: #261
..and prepare for PhpUnit 9 compatibility
*/
private $transactionBatchSize = 1;

public function __construct(EventStore $eventStore, EventListenerInterface $eventListener, Connection $connection)

This comment has been minimized.

Copy link
@bwaidelich

bwaidelich Mar 23, 2020

Author Member

I dislike that we have to pass in the DBAL connection here as it seems unrelated.. But it seems like an OKish compromise for now that can be replaced without major breaking changes hopefully

Copy link
Member

skurfuerst left a comment

great work as usual :)

@bwaidelich bwaidelich merged commit d3d059e into neos:master Mar 23, 2020
1 check passed
1 check passed
continuous-integration/styleci/pr The analysis has passed
Details
@bwaidelich bwaidelich deleted the bwaidelich:bugfix/261-release-highest-applied-sequence-number-during-catchup branch Mar 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

2 participants
You can’t perform that action at this time.