[Improve](streaming job) support postgres partition table sync#60560
[Improve](streaming job) support postgres partition table sync#60560JNSimba merged 3 commits intoapache:masterfrom
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
There was a problem hiding this comment.
Pull request overview
This PR enhances PostgreSQL partition table synchronization in streaming jobs by adding support for publish_via_partition_root parameter when creating publication instances for PostgreSQL 13+. This is a follow-up to PR #59461 which added PostgreSQL synchronization support.
Changes:
- Added support for PostgreSQL partition tables by enabling
publish_via_partition_rootin publications for PostgreSQL 13+ - Enhanced database configuration to support flexible database specification (either in JDBC URL or as a separate parameter)
- Added comprehensive regression test for partition table synchronization including snapshot, incremental changes, and dynamic partition creation
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| test_streaming_postgres_job_partition.groovy | New regression test validating partition table synchronization with insert, update, delete operations and dynamic partition creation |
| test_streaming_postgres_job_partition.out | Expected output for the partition table test showing snapshot and binlog data |
| PostgresSourceReader.java | Enhanced database configuration flexibility and enabled partition table scanning via setIncludePartitionedTables(true) |
| PostgresReplicationConnection.java | Copied and modified Debezium file adding publish_via_partition_root parameter for PostgreSQL 13+ in ALL_TABLES publication mode |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...src/main/java/io/debezium/connector/postgresql/connection/PostgresReplicationConnection.java
Show resolved
Hide resolved
...src/main/java/io/debezium/connector/postgresql/connection/PostgresReplicationConnection.java
Show resolved
Hide resolved
...src/main/java/io/debezium/connector/postgresql/connection/PostgresReplicationConnection.java
Show resolved
Hide resolved
...src/main/java/io/debezium/connector/postgresql/connection/PostgresReplicationConnection.java
Show resolved
Hide resolved
|
run buildall |
TPC-H: Total hot run time: 30509 ms |
ClickBench: Total hot run time: 28.56 s |
|
run external |
|
run vault_p0 |
|
run nonConcurrent |
FE Regression Coverage ReportIncrement line coverage |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
### What problem does this PR solve? Related PR: #59461 To enhance partition table synchronization, add `publish_via_partition_root` when creating a PUBLICATION instance, specifically for PG 13+.
…e#60560) ### What problem does this PR solve? Related PR: apache#59461 To enhance partition table synchronization, add `publish_via_partition_root` when creating a PUBLICATION instance, specifically for PG 13+.
What problem does this PR solve?
Related PR: #59461
To enhance partition table synchronization, add
publish_via_partition_rootwhen creating a PUBLICATION instance, specifically for PG 13+.Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)