branch-4.1: [Improve](streaming job) support custom table name mapping for CDC streaming job #61317#61605
Open
github-actions[bot] wants to merge 1 commit intobranch-4.1from
Open
branch-4.1: [Improve](streaming job) support custom table name mapping for CDC streaming job #61317#61605github-actions[bot] wants to merge 1 commit intobranch-4.1from
github-actions[bot] wants to merge 1 commit intobranch-4.1from
Conversation
…reaming job (#61317) ### What problem does this PR solve? #### Summary Add support for mapping upstream (PostgreSQL) table names to custom downstream (Doris) table names in CDC streaming jobs. Without this feature, the Doris target table must have the same name as the upstream source table. #### New configuration Key format: `"table.<srcTable>.target_table" = "<dstTable>"` in the `FROM` clause properties. ```sql CREATE JOB my_job ON STREAMING FROM POSTGRES ( ... "include_tables" = "pg_orders", "table.pg_orders.target_table" = "doris_orders" ) TO DATABASE mydb (...) ``` When not configured, behavior is unchanged (target table name = source table name). #### Key design decisions - generateCreateTableCmds returns LinkedHashMap<srcName, CreateTableCommand> so callers can distinguish source names (for CDC monitoring) from target names (for DDL) — this fixes a bug where the CDC split assigner would look up the Doris target table name in PostgreSQL - Multi-table merge is supported: two source tables can map to the same Doris table #### Test plan - test_streaming_postgres_job_table_mapping: basic mapping (INSERT/UPDATE/DELETE land in mapped table; Doris table created with target name, not source name) - test_streaming_postgres_job_table_mapping: multi-table merge (two PG tables → one Doris table, snapshot + incremental)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Member
|
run p0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #61317