[source-postgres] : Provide option to advance LSN #34781

akashkulk · 2024-02-02T18:25:16Z

Closes #32946

An option is exposed to set the Debezium property in the setup config : heartbeat.action.query

Set up instructions for users are :

Create a heartbeat table within the database and schema being captured that airbyte has write access to.
Add this table to Airbyte's existing publication.
Set a query that will periodically update that table so as to advance the LSN.

Tested locally. Some notes :

By default, this property is NOT set. No change is expected.
If a user sets a bad query or the airbyte user doesn't have permission to execute the given query, the postgres connector will not fail and continue syncing.

vercel · 2024-02-02T18:25:23Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Feb 8, 2024 7:42pm

github-actions · 2024-02-02T18:25:38Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2024-02-02T18:30:31Z

Coverage report for source-postgres

File	Coverage [46.28%]	❌
PostgresCdcProperties.java	46.28%	❌

Total Project Coverage	69.26%	🍏

rodireich · 2024-02-07T21:40:06Z

...ostgres/src/main/java/io/airbyte/integrations/source/postgres/cdc/PostgresCdcProperties.java

@@ -68,6 +68,11 @@ private static Properties commonProperties(final JdbcDatabase database) {
            : HEARTBEAT_INTERVAL;
    props.setProperty("heartbeat.interval.ms", Long.toString(heartbeatInterval.toMillis()));

+    if (sourceConfig.get("replication_method").has("heartbeat_action_query")
+        && !sourceConfig.get("replication_method").get("heartbeat_action_query").asText().isEmpty()) {
+      props.setProperty("heartbeat.action.query", sourceConfig.get("replication_method").get("heartbeat_action_query").asText());


This seems straightforward configuring an existing debezium property.
I'm just concerned we're opening a door to all sorts of direct sql manipulation.
Not sure how but would be safer to solve it in some templated way so users or some bad actor cannot inject something.

If most of all customers will end up configuring
INSERT INTO airbyte_heartbeat (text) VALUES ('heartbeat')

Can we maybe change field to input something like [schema].[table] and have our code build a query?
could be reasons not to

Discussed offline - I'd like to not make things harder than they are by forcing users to create a specific table. Thinking here is they could just reuse one they already have instead of creating one. WDYT?

rodireich · 2024-02-07T21:40:48Z

...ostgres/src/main/java/io/airbyte/integrations/source/postgres/cdc/PostgresCdcProperties.java

@@ -68,6 +68,11 @@ private static Properties commonProperties(final JdbcDatabase database) {
            : HEARTBEAT_INTERVAL;
    props.setProperty("heartbeat.interval.ms", Long.toString(heartbeatInterval.toMillis()));

+    if (sourceConfig.get("replication_method").has("heartbeat_action_query")


Can we add heartbeat_action_query to a test - even by just configuring it on a test to make sure nothing is broken.

rodireich · 2024-02-07T21:44:19Z

airbyte-integrations/connectors/source-postgres/src/main/resources/spec.json

@@ -289,6 +289,13 @@
                ],
                "default": "After loading Data in the destination",
                "order": 7
+              },
+              "heartbeat_action_query": {


Checking: because this field is optional, existing connection lacking it would just keep working
i.e non-breaking change

Yes - this property is only set if it's populated, otherwise I've verified that the behavior is the same. The default value is empty

Initial commit

4a58f0e

octavia-squidington-iii added area/connectors Connector related issues connectors/source/postgres labels Feb 2, 2024

Merge branch 'master' into akash/pg-advance-lsn

8e3f5c0

vercel bot deployed to Preview February 2, 2024 19:56 View deployment

akashkulk added 3 commits February 2, 2024 11:59

Disable heartbeating (temp)

597079a

Revert heartbeat changes

cbf19c6

Add docs + bump versions

f95500a

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Feb 5, 2024

Merge branch 'master' into akash/pg-advance-lsn

785e8fe

vercel bot deployed to Preview February 5, 2024 18:22 View deployment

Bump expected spec

ea45003

akashkulk marked this pull request as ready for review February 5, 2024 18:42

akashkulk requested a review from a team as a code owner February 5, 2024 18:42

akashkulk requested review from rodireich and stephane-airbyte February 5, 2024 18:42

akashkulk mentioned this pull request Feb 7, 2024

Add heartbeat_action_query parameter #34757

Closed

rodireich reviewed Feb 7, 2024

View reviewed changes

Merge branch 'master' into akash/pg-advance-lsn

cae7ee7

vercel bot deployed to Preview February 8, 2024 18:27 View deployment

Update tests

106ec33

akashkulk requested a review from rodireich February 8, 2024 19:38

rodireich approved these changes Feb 8, 2024

View reviewed changes

akashkulk merged commit e6fff38 into master Feb 8, 2024
25 checks passed

akashkulk deleted the akash/pg-advance-lsn branch February 8, 2024 20:13

xiaohansong pushed a commit that referenced this pull request Feb 13, 2024

[source-postgres] : Provide option to advance LSN (#34781)

eb90e4f

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 21, 2024

[source-postgres] : Provide option to advance LSN (airbytehq#34781)

8c89cc0

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

[source-postgres] : Provide option to advance LSN (airbytehq#34781)

0fbfc41

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

[source-postgres] : Provide option to advance LSN (airbytehq#34781)

0257f39

xiaohansong pushed a commit that referenced this pull request Feb 27, 2024

[source-postgres] : Provide option to advance LSN (#34781)

022d9da

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[source-postgres] : Provide option to advance LSN #34781

[source-postgres] : Provide option to advance LSN #34781

akashkulk commented Feb 2, 2024 •

edited

Loading

vercel bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited by akashkulk

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

rodireich Feb 7, 2024

rodireich Feb 7, 2024

akashkulk Feb 8, 2024

rodireich Feb 7, 2024

rodireich Feb 7, 2024

akashkulk Feb 8, 2024

[source-postgres] : Provide option to advance LSN #34781

[source-postgres] : Provide option to advance LSN #34781

Conversation

akashkulk commented Feb 2, 2024 • edited Loading

vercel bot commented Feb 2, 2024 • edited Loading

github-actions bot commented Feb 2, 2024 • edited by akashkulk Loading

Before Merging a Connector Pull Request

github-actions bot commented Feb 2, 2024 • edited Loading

Coverage report for source-postgres

rodireich Feb 7, 2024

Choose a reason for hiding this comment

rodireich Feb 7, 2024

Choose a reason for hiding this comment

akashkulk Feb 8, 2024

Choose a reason for hiding this comment

rodireich Feb 7, 2024

Choose a reason for hiding this comment

rodireich Feb 7, 2024

Choose a reason for hiding this comment

akashkulk Feb 8, 2024

Choose a reason for hiding this comment

akashkulk commented Feb 2, 2024 •

edited

Loading

vercel bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited by akashkulk

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading