Why
When inserting messages into SqlStreamStore, it is possible to get gaps in the stream, e.g. by a transaction rolling back, or by a transaction not yet ready for commit.
In the first case, the gap is legitimate, but in the second case, any subscribers to the messages should wait until the gap has been filled. It is not possible, upon seeing a gap to canonically determine the scenario under which a gap exists, so SqlStreamStore uses the following algorithm:
- Read a page of messages
- Check that every message's position is monotonically incrementing
- If any message is missing, wait
DefaultReloadIntervalms, and reread a batch
- Return the new batch, without checking for gaps
The reason for ignoring gaps in step 4 is that if the gap is legitimate (e.g. a tx was rolled back), not to ignore gaps would cause the subscriber to continuously reload a page.
The underlying issue is that after the DefaultReloadInterval when a batch will be reloaded, subsequent messages may be written to the message queue. These new messages may also contain gaps, but these new messages will never be checked for inconsistencies.
e.g., consider batch size is 100
First read. 5 Messages in the messages table, batch size is 100, and message 3 has not yet committed (but will be)
ReadAllForwards sees messages [1, 2, *, 4, 5] // Message 3 is missing, so it will reload
Second Read. Now 10 messages from messages table, but message 7 has not been committed (but will be)
ReadAllForwards sees messages [1, 2, 3, 4, 5, 6, *, 8, 9, 10] // Message 7 is missing
ReadAllForwards will not re-validate this batch, and will pass it onto the subscription for processing, permanently hiding message 7 form the subscription.
What
Alter the reloading strategy such that, upon reloading a page due to gaps, the reloaded page is re-validated, taking into account that a message may never exist:
- Load a page
- If a gap is detected:
- Wait for a delay
- Reload the page
- Check for new gaps. Any previously existing gaps can be ignored as abandoned messages (e.g. after a db tx rollback)
- Loop until no new gaps exist
Why
When inserting messages into SqlStreamStore, it is possible to get gaps in the stream, e.g. by a transaction rolling back, or by a transaction not yet ready for commit.
In the first case, the gap is legitimate, but in the second case, any subscribers to the messages should wait until the gap has been filled. It is not possible, upon seeing a gap to canonically determine the scenario under which a gap exists, so SqlStreamStore uses the following algorithm:
DefaultReloadIntervalms, and reread a batchThe reason for ignoring gaps in step 4 is that if the gap is legitimate (e.g. a tx was rolled back), not to ignore gaps would cause the subscriber to continuously reload a page.
The underlying issue is that after the
DefaultReloadIntervalwhen a batch will be reloaded, subsequent messages may be written to the message queue. These new messages may also contain gaps, but these new messages will never be checked for inconsistencies.e.g., consider batch size is 100
First read. 5 Messages in the messages table, batch size is 100, and message 3 has not yet committed (but will be)
Second Read. Now 10 messages from messages table, but message 7 has not been committed (but will be)
What
Alter the reloading strategy such that, upon reloading a page due to gaps, the reloaded page is re-validated, taking into account that a message may never exist: