Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature][CDC] The basic implementation of the CDC source enumerator in the snapshot phase #3253

Closed
3 tasks done
Tracked by #3175
ashulin opened this issue Nov 1, 2022 · 0 comments · Fixed by #3419
Closed
3 tasks done
Tracked by #3175
Assignees
Labels

Comments

@ashulin
Copy link
Member

ashulin commented Nov 1, 2022

Search before asking

  • I had searched in the feature and found no similar feature requirement.

Description

This is a subtask of #3175 to track completion.

Snapshot phase

snapshot-phase

The enumerator generates multiple SnapshotSplits of a table and assigns them to the reader

//  pseudo-code. 
public class SnapshotSplit implements SourceSplit {
    private final String splitId;
    private final TableId tableId;
    private final SeaTunnelRowType splitKeyType;
    private final Object splitStart;
    private final Object splitEnd;
}

When a SnapshotSplit reading is completed, the reader reports the high watermark of the split to the enumerator,
When all SnapshotSplits report high watermark, the enumerator enters the incremental phase.

//  pseudo-code. 
public class CompletedSnapshotSplitReportEvent implements SourceEvent {
    private final String splitId;
    private final Offset highWatermark;
}

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@ashulin ashulin self-assigned this Nov 1, 2022
@ashulin ashulin linked a pull request Nov 14, 2022 that will close this issue
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant