Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the issue of getting initial offset with user provided start #149

Merged
merged 5 commits into from
Aug 14, 2023

Conversation

chaoqin-li1123
Copy link
Contributor

@chaoqin-li1123 chaoqin-li1123 commented Aug 7, 2023

Motivation

Fix the 2 bugs in getUserProvidedMessageId

  1. consumer.asInstanceOf[ConsumerImpl[GenericRecord]].hasMessageAvailable may throw exception if consumer is freshly started.
  2. initial message id is not wrap in UserProvidedMessageId class, which cause the first message to be skipped.
  3. when consumer seek() and read() it actually skip the start message if the message is not a batch message

Modifications

No longer call hasMessageAvailable() method.
Wrap initial offset inside UserProvidedMessageId
change to use reader instead of consumer.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

  • This change is a trivial rework / code cleanup without any test coverage.

  • This change is already covered by existing tests, such as:

  • This change added tests and can be verified as follows:

Documentation

Check the box below.

Need to update docs?

  • doc-required
  • no-need-doc
  • doc

@chaoqin-li1123 chaoqin-li1123 requested review from nlu90 and a team as code owners August 7, 2023 04:45
@github-actions github-actions bot added the no-need-doc This pr does not need any document label Aug 7, 2023
nlu90
nlu90 previously approved these changes Aug 7, 2023
Comment on lines 452 to 453
// Call getLastMessageId to reconnect the consumer before seeking.
if (!consumer.isConnected) consumer.getLastMessageId
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This call seems weird to me. Shouldn't the seek method also handle reconnecting?

@chaoqin-li1123
Copy link
Contributor Author

I find another issue, when consumer seek on offset that is not a batch message id, it will skip a message, previously this method was using reader instead of consumer, which doesn't have the same issue. I have verified this in the test.

@nlu90 nlu90 added this to the 2023-08 milestone Aug 14, 2023
@nlu90 nlu90 merged commit 1840fc1 into streamnative:master Aug 14, 2023
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
no-need-doc This pr does not need any document
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants