Skip to content

Pipe: fix losing points when enable batch mode introduced by progress commit framework refactor#11722

Merged
SteveYurongSu merged 4 commits intoapache:masterfrom
VGalaxies:pipe-lose-points
Dec 19, 2023
Merged

Pipe: fix losing points when enable batch mode introduced by progress commit framework refactor#11722
SteveYurongSu merged 4 commits intoapache:masterfrom
VGalaxies:pipe-lose-points

Conversation

@VGalaxies
Copy link
Copy Markdown
Contributor

@VGalaxies VGalaxies commented Dec 14, 2023

Description

As title. With the introduction of PipeEventCommitter, each DR assigns commit id independently to events. Events with the same commit id in different DRs may be lost if using commit id as a condition for duplication check.


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

@DanielWang2035
Copy link
Copy Markdown
Collaborator

LGTM. The reason is that one connector may transfer events from multiple dataregions, and events from different dataregions may have the same commit id. So we should use equals() instead of comparing commit ids.

@SteveYurongSu SteveYurongSu self-assigned this Dec 15, 2023
@SteveYurongSu SteveYurongSu changed the title Pipe: fix losing points when enable batch mode (WIP) Pipe: fix losing points when enable batch mode introduced by progress commit framework refactor Dec 19, 2023
@SteveYurongSu SteveYurongSu merged commit 82139d4 into apache:master Dec 19, 2023
@VGalaxies VGalaxies deleted the pipe-lose-points branch December 19, 2023 03:47
SzyWilliam pushed a commit to SzyWilliam/iotdb that referenced this pull request Nov 26, 2024
… commit framework refactor (apache#11722)

With the introduction of `PipeEventCommitter`, each DR assigns commit id independently to events. Events with the same commit id in different DRs may be lost if using commit id as a condition for duplication check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants