Add auto_offset_reset option to consumer implementation #266
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This new option allows consumers to handle offset_out_of_range errors gracefully when they occur, rather than simply crashing (see #258). There are three policies to choose from when such an error happens:
These options mimic the Java and Python Kafka libraries for how they handle these errors.
Note: I did not add any tests to this revision, as I was unsure how to approach them. We need a Kafka queue with a retention setting where old messages get deleted so that an offset_out_of_range error gets thrown.
I considered adding a new Kafka queue in the docker setup, though I wanted to get feedback on how to approach that (or even better, if one of the maintainers could help set something up that would be very useful).
I did however test this using real production topics, and as expected it stopped our consumers from crashing all the time.