You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Sep 16, 2021. It is now read-only.
Notification partially processes, producing 100 document events, they all persist (or some portion of them due), but lightblue responds with error or premature EOF
We roll back this notification because we don't know if any of the document events were persisted
Notification is processed again, now we have as many as 200 document events
That being said, they should mostly be superseded / merged away
The text was updated successfully, but these errors were encountered:
The heart of this issue is that we want the notification processing to be transactional.
We are almost there, but can do some more. Brainstorming:
Retrieve new unprocessed notifications or expired processing ones (done)
Update if current to processing (done)
Insert document events as pending w/ source notification ids (not done, inserts as unprocessed)
Update notifications to processed (done)
Update document events to unprocessed (not done, they are already unprocessed)
If 2 fails, stop and start over at 1.
If 3 fails, try to rollback via set notifications to unprocessed and delete any pending document events. Then stop and start over at 1.
If 4 fails, we could retry. It's either processing or processed, both are recoverable. Stop and start over at 1, unless retry is successful. Do not try to rollback notifications and doc events unless you know they are not marked processed, otherwise you could lose doc events.
If 5 fails, we could retry, but that's it. Stop and start over at 1.
For document events, we should retrieve unprocessed (done), old processing (done), as well as expired pending (not done). Expired pending first check if their notifications are processed, and if so these can be processed as well. Otherwise, they can be deleted. We could consider checking if notifications are still processing within some timestamp threshold before deleting. Note that this will probably couple the document event store to the notification store.
Retrieve new unprocessed document events, or expired processing, or expired pending.
If any expired pending, retrieve their notifications. If notifications are processed, keep. If not, drop. (Can potentially keep if notifications still processing, we wouldn't want to delete the result of an in progress notification, but that would mean the notification was processing for at least as long as pending expiration).
Update if current to processing, continue as normal.
alechenninger
changed the title
When we rollback a notification, we can rollback any documentEvents that may have be persisted as a result of partial processing
Make notification processing truly transactional
Jul 8, 2016
alechenninger
changed the title
Make notification processing truly transactional
Make lightblue notification/document event processing truly transactional
Jul 30, 2016
Scenario:
That being said, they should mostly be superseded / merged away
The text was updated successfully, but these errors were encountered: