Skip to content

🎉 Postgres source: retrieve only the tables in the publication under cdc mode - #14447

Merged
Liren Tu (tuliren) merged 14 commits into
masterfrom
liren/check-publication-in-discover
Jul 10, 2022
Merged

🎉 Postgres source: retrieve only the tables in the publication under cdc mode#14447
Liren Tu (tuliren) merged 14 commits into
masterfrom
liren/check-publication-in-discover

Conversation

@tuliren

@tuliren Liren Tu (tuliren) commented Jul 6, 2022

Copy link
Copy Markdown
Contributor

What

@github-actions github-actions Bot added the area/connectors Connector related issues label Jul 6, 2022
@tuliren Liren Tu (tuliren) changed the title Liren/check publication in discover 🎉 Source postgres: retrieve only the tables in the publication under cdc mode Jul 6, 2022
tuliren
Liren Tu (tuliren) commented Jul 6, 2022
View reviewed changes
Comment thread ...rs/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java
}

return stream;
}

@tuliren Liren Tu (tuliren) Jul 6, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods related to CDC catalog manipulation are moved to PostgresCdcCatalogHelper.

tuliren
Liren Tu (tuliren) commented Jul 6, 2022
View reviewed changes
Comment thread ...rs/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java
&& config.get("replication_method").hasNonNull("publication");
LOGGER.info("using CDC: {}", isCdc);
return isCdc;
}

@tuliren Liren Tu (tuliren) Jul 6, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is moved to PostgresUtils.

@tuliren
Liren Tu (tuliren) marked this pull request as ready for review July 6, 2022 13:33
@tuliren

Liren Tu (tuliren) commented Jul 6, 2022 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2623148411
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2623148411
No Python unittests run

Build Passed

Test summary info:

All Passed

@tuliren Liren Tu (tuliren) changed the title 🎉 Source postgres: retrieve only the tables in the publication under cdc mode 🎉 Postgres source: retrieve only the tables in the publication under cdc mode Jul 6, 2022
@tuliren
Liren Tu (tuliren) requested a review from a team as a code owner July 9, 2022 17:45
@tuliren

Liren Tu (tuliren) commented Jul 9, 2022 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2641892558
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2641892558
No Python unittests run

Build Passed

Test summary info:

All Passed

@github-actions github-actions Bot added the area/documentation Improvements or additions to documentation label Jul 10, 2022
@tuliren

Liren Tu (tuliren) commented Jul 10, 2022 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/2643256971


Connector Did it publish? Were definitions generated?
connectors/source-postgres ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@tuliren

Liren Tu (tuliren) commented Jul 10, 2022 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-postgres-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2643257700


Connector Did it publish? Were definitions generated?
connectors/source-postgres-strict-encrypt ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@tuliren
Liren Tu (tuliren) merged commit 078f5fc into master Jul 10, 2022
@tuliren
Liren Tu (tuliren) deleted the liren/check-publication-in-discover branch July 10, 2022 04:54
@octavia-squidington-iii
Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets July 10, 2022 04:55 Inactive
Merged
Merged
@tuliren Liren Tu (tuliren) mentioned this pull request Sep 14, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@subodh1810 Subodh Chaturvedi (subodh1810) Awaiting requested review from subodh1810
@edgao Edward Gao (edgao) Awaiting requested review from edgao

Assignees

No one assigned

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/postgres connectors/source/postgres-strict-encrypt connectors/source/relational-db

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Postgres source: add a check for publications in discover method for CDC

2 participants

@tuliren @octavia-squidington-iii