[cdc-pipeline-connector][doris] Introduce Doris cdc pipeline DataSink #2729
Closed
JNSimba wants to merge 26 commits intoapache:masterfrom
Closed
[cdc-pipeline-connector][doris] Introduce Doris cdc pipeline DataSink #2729JNSimba wants to merge 26 commits intoapache:masterfrom
JNSimba wants to merge 26 commits intoapache:masterfrom
Conversation
added 2 commits
November 22, 2023 11:23
lvyanquan
reviewed
Nov 22, 2023
| private void applyAddColumnEvent(AddColumnEvent event) throws IOException, IllegalArgumentException { | ||
| TableId tableId = event.tableId(); | ||
| List<AddColumnEvent.ColumnWithPosition> addedColumns = event.getAddedColumns(); | ||
| for(AddColumnEvent.ColumnWithPosition col: addedColumns){ |
Contributor
There was a problem hiding this comment.
There are some kinds of ColumnPosition, Does here deal with LAST type only?
Member
Author
There was a problem hiding this comment.
Yes, because adding a value column in Doris does not support adding it to the middle of multiple Key columns.
At the same time, the default is to use JSON format to import. The order of the columns will not affect the data quality. The value column can be directly appended to the end.
added 5 commits
November 23, 2023 14:33
# Conflicts: # flink-cdc-common/src/main/java/com/ververica/cdc/common/schema/Schema.java
Member
Author
|
Addressed the comments. PTAL @lvyanquan |
Contributor
|
@JNSimba Run 'mvn spotless:apply' to fix these violations. |
Member
Author
lvyanquan
reviewed
Nov 27, 2023
Contributor
lvyanquan
reviewed
Nov 27, 2023
| /** A serializer for Event to Tuple2<String, byte[]> */ | ||
| public class DorisEventSerializer implements DorisRecordSerializer<Event> { | ||
| private ObjectMapper objectMapper = new ObjectMapper(); | ||
| private Map<TableId, Schema> schemaMaps = new HashMap<>(); |
Contributor
There was a problem hiding this comment.
A troublesome problem is that we need to maintain the schemaMaps in State to recover from failure, so we need to add a subclass of DorisWriter or DorisBatchWriter to overwrite snapshotState method.
What do you think?
Contributor
There was a problem hiding this comment.
Sorry for this comment, as the discuss before, CreateTableEvent will always sent before DataChangEvent, so we don't need to consider this situation.
Member
Author
There was a problem hiding this comment.
Yes, I understand that it will resend CreateTableEvent even if it is restarted.
# Conflicts: # flink-cdc-common/src/main/java/com/ververica/cdc/common/data/GenericStringData.java
lvyanquan
reviewed
Nov 28, 2023
added 2 commits
November 28, 2023 21:52
# Conflicts: # flink-cdc-common/src/main/java/com/ververica/cdc/common/utils/SchemaUtils.java
lvyanquan
reviewed
Nov 29, 2023
added 4 commits
November 29, 2023 16:30
# Conflicts: # flink-cdc-common/src/main/java/com/ververica/cdc/common/utils/SchemaUtils.java
Member
Author
|
Addressed the comments. PTAL @lvyanquan |
Contributor
|
Thanks for your contribution. Over look good to me. Left some comments, and can you clean the commit message？ |
lvyanquan
reviewed
Nov 30, 2023
lvyanquan
reviewed
Nov 30, 2023
lvyanquan
reviewed
Nov 30, 2023
Member
Author
|
Addressed the comments. PTAL @lvyanquan |
lvyanquan
reviewed
Dec 1, 2023
Member
Author
|
Addressed the comments. PTAL @lvyanquan |
JNSimba
changed the title
Contributor
|
Resolved by 4abd86a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This closes #2646
add a
DorisDataSinkthat implement interface ofDataSinkto build a pipeline.