DATAREDIS-1230 - Revise error handling in StreamReceiver and StreamMessageListenerContainer #576
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ssageListenerContainer. Reading and deserialization in StreamReceiver and StreamMessageListener is now decoupled from each other to allow fine-grained control over errors and resumption. Previously, we used the Template API to read and deserialize stream records. Now the actual read happens before the deserialization so that errors during deserialization of individual messages can be handled properly. This split also allows advancing in the stream read. Previously, the failed deserialization prevented of getting hold of the non-serialized Stream record which caused the stream receiver to remain at the offset that fetched the offending record which effectively lead to an infinite loop. We also support a resume function in StreamReceiver to control whether stream reads should be resumed or terminated.
christophstrobl
pushed a commit
that referenced
this pull request
Dec 7, 2020
…ssageListenerContainer. Reading and deserialization in StreamReceiver and StreamMessageListener is now decoupled from each other to allow fine-grained control over errors and resumption. Previously, we used the Template API to read and deserialize stream records. Now the actual read happens before the deserialization so that errors during deserialization of individual messages can be handled properly. This split also allows advancing in the stream read. Previously, the failed deserialization prevented of getting hold of the non-serialized Stream record which caused the stream receiver to remain at the offset that fetched the offending record which effectively lead to an infinite loop. We also support a resume function in StreamReceiver to control whether stream reads should be resumed or terminated. Original Pull Request: #576
christophstrobl
pushed a commit
that referenced
this pull request
Dec 7, 2020
…ssageListenerContainer. Reading and deserialization in StreamReceiver and StreamMessageListener is now decoupled from each other to allow fine-grained control over errors and resumption. Previously, we used the Template API to read and deserialize stream records. Now the actual read happens before the deserialization so that errors during deserialization of individual messages can be handled properly. This split also allows advancing in the stream read. Previously, the failed deserialization prevented of getting hold of the non-serialized Stream record which caused the stream receiver to remain at the offset that fetched the offending record which effectively lead to an infinite loop. We also support a resume function in StreamReceiver to control whether stream reads should be resumed or terminated. Original Pull Request: #576
This was referenced Dec 30, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reading and deserialization in StreamReceiver and StreamMessageListener is now decoupled from each other to allow fine-grained control over errors and resumption.
Previously, we used the Template API to read and deserialize stream records. Now the actual read happens before the deserialization so that errors during deserialization of individual messages can be handled properly. This split also allows advancing in the stream read. Previously, the failed deserialization prevented of getting hold of the non-serialized Stream record which caused the stream receiver to remain at the offset that fetched the offending record which effectively lead to an infinite loop.
We also support a resume function in StreamReceiver to control whether stream reads should be resumed or terminated.
Related ticket: DATAREDIS-1230.