-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[pipeline-connector] Ability to provide a schema replacement #2908
base: master
Are you sure you want to change the base?
[pipeline-connector] Ability to provide a schema replacement #2908
Conversation
7c9ff13
to
6e21b89
Compare
6e21b89
to
e1e246e
Compare
PTAL @banmoy |
Hi, @WholeWorld-Timothy this idea looks good to me. |
The current mode is a support for modification mode, and I think it is a good idea to add a table name prefix or a table name suffix based on this placeholder, so I can try it out. |
e1e246e
to
2ca1150
Compare
PTAL @lvyanquan |
TableId.parse( | ||
replaceBy.getNamespace(), | ||
replaceBy.getSchemaName(), | ||
replaceBy.getTableName().replace("<>", tableId.getTableName())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to extract this literal into a static constant, and determine a value that will not be accidentally used in the table name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed a version and put the definition of the <> symbol into the configuration file, just like this form:
route:
- source-table: mrtdb.[\S]*
sink-table: mrtdb_sd.sd_s<>
replace-symbol: s<>
In this way, the user can define the replacement symbols by themselves. Is it any better?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1. Please add some tests for these scenes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two methods, RouteFunctionTest#testSchemaChangeRouting And RouteFunctionTest#testTableNameChangeRouting were modified to test this scenes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To expose this parameter to the users, please add description to cdc-pipeline.md.
And add this configuration to testParsingFullDefinition.
2ca1150
to
fb23579
Compare
Signed-off-by: 张田 <zhang_tian@inspur.com>
Signed-off-by: 张田 <zhang_tian@inspur.com>
…y themselves. Signed-off-by: 张田 <zhang_tian@inspur.com>
Signed-off-by: 张田 <zhang_tian@inspur.com>
fb23579
to
87f73cf
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
My local compilation and testing are correct, so the error in check may not be caused by the code? How can rerun these check ? I click rerun in github desktop doesn t work |
Thanks for your contribution, please rebase the master branch and I'll take a look on the CI failure later. Besides, there are some doc modifications which seems to be unnecessary, could you please revert them back? |
Hi @WholeWorld-Timothy, could you please rebase this PR with latest cc @whhe |
In our usage scenario, the source schema of mysql and the target schema of starrocks are often different, but the table names and column names are consistent, and this pr provides this capability.
Maintain a route configuration as follows:
Represents the table below the test will be transferred to the test02, where the symbol <> the representative table name is unchanged, just change the schema. For example, the test.table would be replaced with the test02.table.
And maintain a route configuration as follows:
Represents the table below the test will be transferred to the test02, Replace the original table name with the <> symbol. For example, the test.table would be replaced with the test02.s02_table.
And the user can define the replacement symbols by themselves as follows:
Represents the table below the test will be transferred to the test02, Replace the original table name with the s<> symbol.