Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip control records without auto-resolve #511

Merged
merged 4 commits into from
Oct 1, 2019

Conversation

tulios
Copy link
Owner

@tulios tulios commented Sep 30, 2019

This PR solves issue #403.

It does two things:

  1. Automatically resolve the control record offset when resolving the last non-filtered offset of the batch
  2. When the batch is an empty control batch, automatically resolve the offset

I wrote more on the code documentation; I will paste here for documentation purposes.

The transactional producer generates a control record after committing the transaction.
The control record is the last record on the RecordBatch, and it is filtered before it
reaches the eachBatch callback. When disabling auto-resolve, the user-land code won't
be able to resolve the control record offset, since it never reaches the callback,
causing stuck consumers as the consumer will never move the offset marker.

When the last offset of the batch is resolved, we should automatically resolve
the control record offset as this entry doesn't have any meaning to the user-land code,
and won't interfere with the stream processing.

I am pinging the people in the original issue since it's a bit old and you might miss this PR.
@kkurten @drojas @JaapRood @jasine @brandonl

This commit does two things:

1) Automatically resolve the control record offset when resolving the 
last non-filtered offset of the batch

2) When the batch is an empty control batch, automatically resolve the 
offset
@tulios tulios requested a review from Nevon September 30, 2019 21:49
Copy link
Collaborator

@Nevon Nevon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We talked about a potential way of testing this better, either by using the request instrumentation event to see how many fetches we make (might be flaky), or by setting the maxBytes low enough that you only ever get a single message in each batch, and then publishing a message after finishing the transaction. That way you should get "stuck" on the control batch without the fix.

@JaapRood
Copy link
Collaborator

JaapRood commented Oct 1, 2019

Seems like a totally reasonable way to address this issue 👌👍

@Nevon Nevon merged commit fcf6435 into master Oct 1, 2019
@Nevon Nevon deleted the skip-control-record-without-auto-resolve branch October 1, 2019 13:17
@tulios
Copy link
Owner Author

tulios commented Oct 1, 2019

Pre-release 1.12.0-beta.0 published with the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants