Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix data duplicate and data loss #154

Merged
merged 1 commit into from
Aug 14, 2023

Conversation

chaoqin-li1123
Copy link
Contributor

@chaoqin-li1123 chaoqin-li1123 commented Aug 12, 2023

Motivation

When start offset is exclusive and equal to end offset, current code will read past the end, which results in data duplicate.
Reader seek() is redundant inside pulsar RDD(because start offset is already specified for the reader) and sometimes cause data loss(sometimes the cursor forward when seeking the current offset).

Modifications

Remove redundant reader seek()
Set isLast flag to true if start offset = end offset when start offset is exclusive.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

  • This change is a trivial rework / code cleanup without any test coverage.

  • This change is already covered by existing tests, such as:

  • This change added tests and can be verified as follows:

Documentation

Check the box below.

Need to update docs?

  • doc-required
  • no-need-doc
  • doc

@chaoqin-li1123 chaoqin-li1123 requested review from nlu90 and a team as code owners August 12, 2023 05:18
@github-actions github-actions bot added the no-need-doc This pr does not need any document label Aug 12, 2023
@nlu90 nlu90 added this to the 2023-08 milestone Aug 14, 2023
@nlu90 nlu90 merged commit 0081e16 into streamnative:master Aug 14, 2023
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
no-need-doc This pr does not need any document
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants