Skip to content

Conversation

@sandynz
Copy link
Contributor

@sandynz sandynz commented Jun 1, 2023

Related to #22500

Reduce CDC code layer, make it similar as migration etc.

Purpose: 1) Cleaner code, 2) Prepare for unified code refactoring.

Changes proposed in this pull request:

  • Add unified PipelineSink and impl: PipelineDataSourceSink and PipelineSocketSink
  • Add CDCImporter, CDCAckId; Remove SocketSinkImporter, SocketSinkImporterCreator, CDCAckHolder, CDCDataRecordUtils, DataRecordComparatorGenerator; Refactor PipelineChannel, PipelineSink and impls;
  • Add SingleChannelConsumerImporter for common usage
  • Add inventory and incremental task for CDC; Refactor CDCJobPreparer, CDCJob, CDCTasksRunner

Part of general structure and process:

  • Use 1 job item of ejob, and sharding items are simulated by CDCJob, since task progress is binded in PipelineJobItemContext.
  • 1 importer will consume several dumpers for inventory and incremental tasks.
  • Incremental tasks will start after all inventory tasks are completed.

Before committing this PR, I'm sure that I have checked the following options:

  • My code follows the code of conduct of this project.
  • I have self-reviewed the commit code.
  • I have (or in comment I request) added corresponding labels for the pull request.
  • I have passed maven check locally : ./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e.
  • I have made corresponding changes to the documentation.
  • I have added corresponding unit tests for my changes.

sandynz added 30 commits May 24, 2023 15:12
…temContext to PipelineTask, update related classes
Remove SocketSinkImporter, SocketSinkImporterCreator, CDCAckHolder, CDCDataRecordUtils, DataRecordComparatorGenerator;
Refactor PipelineChannel, PipelineSink and impls;
@sandynz sandynz marked this pull request as ready for review June 2, 2023 07:26
@azexcy azexcy merged commit 2d7bc66 into apache:master Jun 2, 2023
@sandynz sandynz deleted the refactorCdcLayer branch June 2, 2023 10:22
@sandynz
Copy link
Contributor Author

sandynz commented Jun 2, 2023

TODO: Improve XA support in next PR

@sandynz sandynz restored the refactorCdcLayer branch June 2, 2023 10:38
@sandynz sandynz deleted the refactorCdcLayer branch June 3, 2023 06:52
@sandynz sandynz mentioned this pull request Jun 3, 2023
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants