feat(core): Add partitioned table support #410

iambriccardo · 2025-10-24T07:22:54Z

This PR adds partitioned table support by treating a partitioned table as a single entity to replicate.

Implemented Behavior

Partition detachment handling:

If a partition is detached, the downstream data will NOT be deleted and the table will stop replication.
If a partition is detached but the publication includes ALL TABLES (with or without schema selection), the detached table will be added as a standalone table when the pipeline restarts.

publish_via_partition_root handling:

If publish_via_partition_root=false the system will throw an error if there is at least one partitioned table in the publication.
If publish_via_partition_root=true the system behaves as expected, treating the partitioned tables as one big table.

Testing

Several tests have been added to verify the behavior functions correctly.

Requirements

Note: FOR TABLES IN [schema] is only supported from Postgres 15+

coveralls · 2025-10-24T07:31:16Z

Pull Request Test Coverage Report for Build 18839109103

Details

262 of 271 (96.68%) changed or added relevant lines in 6 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 82.469%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
etl-api/src/db/publications.rs	0	1	0.0%
etl/src/pipeline.rs	19	21	90.48%
etl/src/test_utils/event.rs	10	12	83.33%
etl/src/replication/client.rs	167	171	97.66%

Totals
Change from base Build 18773857928:	0.2%
Covered Lines:	15444
Relevant Lines:	18727

💛 - Coveralls

imor

A few questions:

When publish_via_partition_root is false, do we copy the root table as well the the partitions?
If publish_via_partition_root is true, and a new partition is added, do we replicate it successfully? We don't have tests for this case.

imor · 2025-10-27T05:00:07Z

docs/how-to/configure-postgres.md

+
+```sql
+-- Create publication with partitioned table support
+CREATE PUBLICATION my_publication FOR TABLE users, orders WITH (publish_via_partition_root = true);


Nit: user lowercase SQL.

Why would we want to have lowercase in user facing docs? Most docs have the uppercase convention and I felt like it would be better to follow their lead.

Our existing docs have mostly lowercase SQL. You can check it out in the docs folder.

etl/src/replication/client.rs

imor · 2025-10-27T06:12:51Z

etl/tests/pipeline_with_partitioned_table.rs

+        .await
+        .unwrap();
+
+    let _ = pipeline.shutdown_and_wait().await;


To test for an absence of events, we should insert one row into an existing partition after detaching & dropping the partition table and wait for that event to arrive. In the current form, the test could be passing because we shutdown the pipeline too quickly.

Yep, that's a fair comment. I will see what I can do.

imor · 2025-10-27T06:13:59Z

etl/tests/pipeline_with_partitioned_table.rs

+        .await
+        .unwrap();
+
+    let _ = pipeline.shutdown_and_wait().await;


iambriccardo · 2025-10-27T08:50:36Z

A few questions:

When publish_via_partition_root is false, do we copy the root table as well the the partitions?

If publish_via_partition_root is true, and a new partition is added, do we replicate it successfully? We don't have tests for this case.

We do copy only the root table, meaning all the data across all partitions. The difference is that messages in the stream are tagged with partition oids instead of root oid, meaning that they will be skipped.
partitioned_table_copy_and_streams_new_data_from_new_partition already exists.

iambriccardo · 2025-10-27T09:44:23Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2025-10-27T09:49:09Z

etl/src/replication/client.rs

    /// Retrieves the OIDs of all tables included in a publication.
+    ///
+    /// For partitioned tables with `publish_via_partition_root=true`, this returns only the parent
+    /// table OID. The query uses a recursive CTE to walk up the partition inheritance hierarchy
+    /// and identify root tables that have no parent themselves.
    pub async fn get_publication_table_ids(
        &self,
        publication_name: &str,
    ) -> EtlResult<Vec<TableId>> {
-        let publication_query = format!(
-            "select c.oid from pg_publication_tables pt 
-         join pg_class c on c.relname = pt.tablename 
-         join pg_namespace n on n.oid = c.relnamespace AND n.nspname = pt.schemaname 
-         where pt.pubname = {};",
-            quote_literal(publication_name)
+        let query = format!(
+            r#"
+            with recursive pub_tables as (
+                -- Get all tables from publication (pg_publication_tables includes explicit tables,
+                -- ALL TABLES publications, and FOR TABLES IN SCHEMA publications)
+                select c.oid
+                from pg_publication_tables pt
+                join pg_class c on c.relname = pt.tablename
+                join pg_namespace n on n.oid = c.relnamespace and n.nspname = pt.schemaname
+                where pt.pubname = {pub}
+            ),
+            hierarchy(relid) as (
+                -- Start with published tables
+                select oid from pub_tables
+
+                union
+
+                -- Recursively find parent tables in inheritance hierarchy
+                select i.inhparent
+                from pg_inherits i
+                join hierarchy h on h.relid = i.inhrelid
+            )
+            -- Return only root tables (those without a parent)
+            select distinct relid as oid
+            from hierarchy
+            where not exists (
+                select 1 from pg_inherits i where i.inhrelid = hierarchy.relid
+            );
+            "#,
+            pub = quote_literal(publication_name)
        );

-        let mut table_ids = vec![];
-        for msg in self.client.simple_query(&publication_query).await? {
+        let mut roots = vec![];
+        for msg in self.client.simple_query(&query).await? {
            if let SimpleQueryMessage::Row(row) = msg {
-                // For the sake of simplicity, we refer to the table oid as table id.
                let table_id = Self::get_row_value::<TableId>(&row, "oid", "pg_class").await?;
-                table_ids.push(table_id);
+                roots.push(table_id);
            }
        }

-        Ok(table_ids)
+        Ok(roots)


Collapse child partitions even when publication publishes child OIDs

The new get_publication_table_ids query now always walks up pg_inherits and returns only root tables. When the publication was created with publish_via_partition_root = false (the PostgreSQL default), logical replication messages still contain the child partition OIDs. Because only the parent ID is returned here, the schema cache never contains entries for those child OIDs and handle_relation_message will raise MissingTableSchema as soon as a child relation message arrives (replication/apply.rs handle_relation_message). This turns what used to be a “no CDC after copy” scenario into a hard pipeline failure. Either avoid collapsing when pubviaroot is false or teach the apply loop to handle child OIDs gracefully.

Useful? React with 👍 / 👎.

I have updated the code to now validate whether a publication has partitioned tables and if publish_via_partition_root=false the pipeline won't start.

imor · 2025-10-27T11:12:59Z

A few questions:

When publish_via_partition_root is false, do we copy the root table as well the the partitions?

If publish_via_partition_root is true, and a new partition is added, do we replicate it successfully? We don't have tests for this case.

We do copy only the root table, meaning all the data across all partitions. The difference is that messages in the stream are tagged with partition oids instead of root oid, meaning that they will be skipped.

Okay, does this mean only initial table copy will be done and no CDC on the root?

partitioned_table_copy_and_streams_new_data_from_new_partition already exists.

Cool, that's great.

iambriccardo · 2025-10-27T11:16:34Z

A few questions:

When publish_via_partition_root is false, do we copy the root table as well the the partitions?

If publish_via_partition_root is true, and a new partition is added, do we replicate it successfully? We don't have tests for this case.

We do copy only the root table, meaning all the data across all partitions. The difference is that messages in the stream are tagged with partition oids instead of root oid, meaning that they will be skipped.

Okay, does this mean only initial table copy will be done and no CDC on the root?

partitioned_table_copy_and_streams_new_data_from_new_partition already exists.

Cool, that's great.

Exactly, that's the result. Since it's a weird behavior, I have added an additional check on startup that fails the pipeline when the setting is on false and you have at least one partitioned table.

imor · 2025-10-27T11:19:00Z

A few questions:

When publish_via_partition_root is false, do we copy the root table as well the the partitions?

If publish_via_partition_root is true, and a new partition is added, do we replicate it successfully? We don't have tests for this case.

We do copy only the root table, meaning all the data across all partitions. The difference is that messages in the stream are tagged with partition oids instead of root oid, meaning that they will be skipped.

Okay, does this mean only initial table copy will be done and no CDC on the root?

partitioned_table_copy_and_streams_new_data_from_new_partition already exists.

Cool, that's great.

Exactly, that's the result. Since it's a weird behavior, I have added an additional check on startup that fails the pipeline when the setting is on false and you have at least one partitioned table.

Alright, that should work. We can refine this behaviour in future.

iambriccardo added 7 commits October 21, 2025 12:42

feat(core): Add partitioned table support

282b0e6

Improve

0a57cfe

Improve

cd0b607

Improve

52fd049

Improve

8624c3a

Improve

f2378ff

Improve

ac65190

iambriccardo changed the title ~~riccardobusetti/etl 268 partitioned tables do not work directly due to lack of pk~~ @iambriccardo feat(core): Add partitioned table support Oct 24, 2025

iambriccardo changed the title ~~@iambriccardo feat(core): Add partitioned table support~~ feat(core): Add partitioned table support Oct 24, 2025

iambriccardo mentioned this pull request Oct 24, 2025

feat(postgres): add partitioned table support #297

Closed

iambriccardo added 7 commits October 24, 2025 10:35

Improve

8d553c3

Improve

319273b

Improve

29f1f41

Improve

9a4d1fc

Improve

bbfdf95

Improve

65f0b52

Improve

12416f4

iambriccardo force-pushed the riccardobusetti/etl-268-partitioned-tables-do-not-work-directly-due-to-lack-of-pk branch from f5e3b11 to 12416f4 Compare October 24, 2025 13:32

iambriccardo added 3 commits October 24, 2025 15:48

Improve

2127af1

Improve

a51a8d1

Improve

c87820f

iambriccardo marked this pull request as ready for review October 24, 2025 14:52

iambriccardo requested a review from a team as a code owner October 24, 2025 14:52

iambriccardo added 3 commits October 24, 2025 17:12

Improve

5276bec

Improve

0ac142a

Improve

3fec642

imor reviewed Oct 27, 2025

View reviewed changes

Improve

bf4ef9f

iambriccardo requested a review from imor October 27, 2025 09:43

chatgpt-codex-connector bot reviewed Oct 27, 2025

View reviewed changes

iambriccardo added 4 commits October 27, 2025 11:31

Improve

dbede95

Improve

899ef69

Improve

fb43b9d

Improve

df3044a

Improve

17b0357

imor approved these changes Oct 27, 2025

View reviewed changes

iambriccardo merged commit 17b2e1d into main Oct 27, 2025
9 checks passed

iambriccardo deleted the riccardobusetti/etl-268-partitioned-tables-do-not-work-directly-due-to-lack-of-pk branch October 27, 2025 11:45

Uh oh!

Uh oh!

feat(core): Add partitioned table support #410

feat(core): Add partitioned table support #410

Uh oh!

Conversation

iambriccardo commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implemented Behavior

Testing

Requirements

Uh oh!

coveralls commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 18839109103

Details

💛 - Coveralls

Uh oh!

imor left a comment

Choose a reason for hiding this comment

Uh oh!

imor Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

iambriccardo Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

imor Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

imor Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

iambriccardo Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

imor Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

iambriccardo commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iambriccardo commented Oct 27, 2025

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

iambriccardo Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

imor commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iambriccardo commented Oct 27, 2025

Uh oh!

imor commented Oct 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

iambriccardo commented Oct 24, 2025 •

edited

Loading

coveralls commented Oct 24, 2025 •

edited

Loading

iambriccardo commented Oct 27, 2025 •

edited

Loading

iambriccardo Oct 27, 2025 •

edited

Loading

imor commented Oct 27, 2025 •

edited

Loading