Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reader: allow config to return OffsetOutOfRange errors #917

merged 4 commits into from Jun 3, 2022


Copy link

@dominicbarnes dominicbarnes commented May 20, 2022

This PR aims to change the handling of OffsetOutOfRange errors in the Reader, but includes some quirks to retain backwards-compatibility. This will fix #891 and makes #892 unnecessary.

In short, when a topic is re-created (ie: deleted and re-created with same name), any readers that are already running will begin encountering OffsetOutOfRange errors, but will retry them indefinitely, blocking the program from making progress. This error can also occur when a particular broker advances past the leader, so retrying made sense as this was usually temporary.

The original issue and PR related to this opted to automatically "reset" the offset when this error was encountered, but this felt a little too magical and I've opted instead to return an error to allow the client to handle it in whatever way makes sense for the use-case. Since topic re-creation feels like an edge-case (such as only happening during active development), it feels best to be more explicit rather than adopting this logic into the reader itself.

A configuration flag is being added, largely to allow existing clients to remain unaffected by these changes, with the expectation that this behavior should change in a future release of kafka-go.

Copy link

@achille-roussel achille-roussel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great 🙌 thanks for the contribution!

@achille-roussel achille-roussel self-assigned this May 27, 2022
@dominicbarnes dominicbarnes merged commit da91759 into main Jun 3, 2022
@dominicbarnes dominicbarnes deleted the handle-offset-out-of-range branch June 3, 2022 16:24
Copy link

wwade commented Jun 4, 2022

@dominicbarnes thanks for taking this on, much appreciated. I rolled it in to my project and ran my test again and it worked nicely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

reader doesn't recover (quickly) from OffsetOutOfRange "Offset Out Of Range"
3 participants