Skip to content

Conversation

@thecodeboss
Copy link

@thecodeboss thecodeboss commented Dec 14, 2017

This new option allows consumers to handle offset_out_of_range errors gracefully when they occur, rather than simply crashing (see #258). There are three policies to choose from when such an error happens:

  1. Move consumer offset to the earliest available
  2. Move consumer offset to the latest available
  3. (default) let it crash
    These options mimic the Java and Python Kafka libraries for how they handle these errors.

Note: I did not add any tests to this revision, as I was unsure how to approach them. We need a Kafka queue with a retention setting where old messages get deleted so that an offset_out_of_range error gets thrown.
I considered adding a new Kafka queue in the docker setup, though I wanted to get feedback on how to approach that (or even better, if one of the maintainers could help set something up that would be very useful).
I did however test this using real production topics, and as expected it stopped our consumers from crashing all the time.

mix.exs Outdated
[
app: :kafka_ex,
version: "0.8.1",
version: "0.8.2",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version bump is handled when we do a release, so this part should be removed.

@dantswain
Copy link
Collaborator

Thanks, @thecodeboss, this looks pretty good! Just a couple minor things:

  • Can you please take a look and see if the new option needs to be added to any @types or @specs?
  • Remove the version bump. I'll likely be doing another release in the next couple days and I'll wrap this up into that one. This also probably deserves a minor version bump since it adds new functionality.

@thecodeboss
Copy link
Author

thecodeboss commented Dec 17, 2017

Thanks for the feedback @dantswain and @joshuawscott - I undid the version bump and added the new option to @type option. I looked pretty thoroughly, seems to be the only one.

@thecodeboss
Copy link
Author

@dantswain bump, would like to get this merged

@dantswain dantswain merged commit 0aad419 into kafkaex:master Jan 19, 2018
@dantswain
Copy link
Collaborator

@thecodeboss Sorry about the delay. I've been really busy.

robotarmy pushed a commit to RAM9/kafka_ex that referenced this pull request Apr 7, 2025
Add auto_offset_reset option to consumer implementation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants