[Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over by nguyenaiden · Pull Request #30270 · airbytehq/airbyte

Duy Nguyen (nguyenaiden) · 2023-09-07T23:22:49Z

What

Utilize Primary keys for initial sync then switch over to user defined cursor once high water mark is found and persisted in state.

How

With normal incremental syncs now, we will be using Primary key for the initial sync and retrieve the high water mark in the beginning and construct all of the relevant Primary Key and Standard Sync iterators at the beginning of the sync.

This includes a SELECT MAX query to retrieve max cursor value and populate the initial and final StreamState to provide a checkpoint so that there won't be any missing records on subsequent syncs.

Note: The getIncrementalIterators method in the MysqlInitialLoadHandler class now has logic to handle the edge case of when the customer de-select a primary key column. Solution here is to re-add the column names to the selectedDatabaseFields in order to construct the correct SELECT query, but those columns will not be part of the record as the CatalogHelpers.getTopLevelFieldNames(airbyteStream) will not include the de-selected fields.

Tests with flag turned on is in the file MySqlPkJdbcSourceAcceptanceTest.java

github-actions · 2023-09-07T23:23:16Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

Akash Kulkarni (akashkulk) · 2023-09-08T16:25:23Z

Why is a change to this file needed?

Akash Kulkarni (akashkulk) · 2023-09-08T16:36:10Z

We should escape quote cursor fields like in this PR : https://github.com/airbytehq/airbyte/pull/30059/files

Akash Kulkarni (akashkulk) · 2023-09-08T16:40:28Z

Why are we deleting this code?

I moved this to the Utils class so I can use it as well

can we move this back now that this is only being used in here! Just trying to minimise the changes.

I'm good with that! On it

Akash Kulkarni (akashkulk) · 2023-09-08T16:43:44Z

Can we move this to MySqlInitialReadUtil? I assume this is the code that is doing the main logic of figuring out which streams to sync via pk?

Moved it! I actually eliminated the filtering based on state_type = cursor_based since anything that's not PK is implicitly cursor_based.

Akash Kulkarni (akashkulk) · 2023-09-08T16:57:49Z

This is similar to the PostgresCursorBasedStateManager class correct? Basically just to enable writing the state_type and version keys? Can we add a comment for that

IIRC we had a ticket to refactor this in StreamStateManager when we implemented source-mysql - do you think we should do that here?

Akash Kulkarni (akashkulk) · 2023-09-08T17:28:12Z

can we remove this and just keep InitialLoadStreams? Anything that is not in InitialLoadStreams would implicitly be a cursor based streams right?

Yep! Addressed above.

github-actions · 2023-09-11T05:00:24Z

source-snowflake test report (commit `569bfc0623`) - ❌

⏲️ Total pipeline duration: 13mn02s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-snowflake/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-snowflake test

github-actions · 2023-09-11T05:02:42Z

Coverage report for source-postgres

File	Coverage [87.89%]	🍏
PostgresSource.java	88.62%	🍏
PostgresUtils.java	84.9%	🍏

Total Project Coverage	70.56%	🍏

github-actions · 2023-09-11T15:03:11Z

source-cockroachdb test report (commit `569bfc0623`) - ❌

⏲️ Total pipeline duration: 602mn36s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-cockroachdb docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-cockroachdb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-cockroachdb test

github-actions · 2023-09-11T15:44:43Z

source-mysql-strict-encrypt test report (commit `569bfc0623`) - ❌

⏲️ Total pipeline duration: 41mn21s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

github-actions · 2023-09-11T16:06:26Z

source-oracle-strict-encrypt test report (commit `569bfc0623`) - ❌

⏲️ Total pipeline duration: 21mn31s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-oracle-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-oracle-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-oracle-strict-encrypt test

github-actions · 2023-09-11T16:36:03Z

source-postgres test report (commit `569bfc0623`) - ✅

⏲️ Total pipeline duration: 29mn26s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-09-18T21:52:38Z

source-mysql-strict-encrypt test report (commit `76c62cea72`) - ❌

⏲️ Total pipeline duration: 02mn58s

Step	Result
Build connector tar	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

github-actions · 2023-09-18T21:55:37Z

source-mysql test report (commit `76c62cea72`) - ✅

⏲️ Total pipeline duration: 02mn36s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-18T21:58:28Z

source-postgres test report (commit `76c62cea72`) - ❌

⏲️ Total pipeline duration: 02mn31s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-09-18T21:58:55Z

source-mssql test report (commit `76c62cea72`) - ❌

⏲️ Total pipeline duration: 13.21s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql test

WIP WIP Functional feature done

Formatting

Duy Nguyen (nguyenaiden) · 2023-09-18T22:16:18Z

/approve-and-merge reason="ci passes and features are gated behind flag". Source-postgres and mssql changes are non-functional. "

Octavia The Approver (octavia-approvington) · 2023-09-18T22:16:58Z

Jerry would be proud

github-actions · 2023-09-18T22:43:20Z

source-mysql test report (commit `b0c1071caf`) - ❌

⏲️ Total pipeline duration: 30mn51s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-18T22:43:50Z

source-mssql test report (commit `b0c1071caf`) - ❌

⏲️ Total pipeline duration: 6.00s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql test

github-actions · 2023-09-18T22:46:26Z

source-postgres test report (commit `b0c1071caf`) - ❌

⏲️ Total pipeline duration: 01mn51s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-09-18T22:55:26Z

source-mysql-strict-encrypt test report (commit `b0c1071caf`) - ❌

⏲️ Total pipeline duration: 08mn25s

Step	Result
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues CDK Connector Development Kit connectors/source/mysql connectors/source/jdbc labels Sep 7, 2023

Akash Kulkarni (akashkulk) reviewed Sep 8, 2023

View reviewed changes

Comment thread .../connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java Outdated

Akash Kulkarni (akashkulk) reviewed Sep 8, 2023

View reviewed changes

Octavia Squidington III (octavia-squidington-iii) added connectors/source/cockroachdb connectors/source/oracle connectors/source/postgres connectors/source/snowflake connectors/source/mysql-strict-encrypt connectors/source/oracle-strict-encrypt labels Sep 11, 2023

Duy Nguyen (nguyenaiden) requested a review from Subodh Chaturvedi (subodh1810) September 11, 2023 04:44

Duy Nguyen (nguyenaiden) marked this pull request as ready for review September 11, 2023 04:45

Duy Nguyen (nguyenaiden) requested a review from a team as a code owner September 11, 2023 04:45

Duy Nguyen (nguyenaiden) requested a review from a team September 11, 2023 04:45

Duy Nguyen (nguyenaiden) changed the title ~~[Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over [WIP]~~ [Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over Sep 11, 2023

Duy Nguyen (nguyenaiden) requested a review from Rodi Reich Zilberman (rodireich) September 11, 2023 04:45

nguyenaiden added 14 commits September 18, 2023 15:06

WIP standard sync

532bc9a

WIP WIP Functional feature done

uncomment test

6b7d05b

Separated out cursor and pk streams

0527fa1

Feature complete

bb67fb0

Added necessary override decorators and removed unneeded null checks

f7583ae

Addressed comments

39f43dd

Added override back to Oracle test

aa17ddb

Localize changes to just MysqlPkJdbcSourceAcceptanceTests

58a6e39

Formatting

Reverted changes to cockroachdb test

e0bbf87

Bumped metadata and update changelog

8dfdd4c

Update getStateData override to not use legacy states

4eedbf9

Bumped dockerfile

3c31b58

Added pk test for strict encrypt as well

4a3df49

Bump mysql version after flag has been removed

b0c1071

Duy Nguyen (nguyenaiden) force-pushed the mysql-standard-pk-sync branch from 372e55d to b0c1071 Compare September 18, 2023 22:07

Octavia Squidington III (octavia-squidington-iii) removed the area/documentation Improvements or additions to documentation label Sep 18, 2023

Octavia The Approver (octavia-approvington) approved these changes Sep 18, 2023

View reviewed changes

Octavia The Approver (octavia-approvington) merged commit f226503 into master Sep 18, 2023

Octavia The Approver (octavia-approvington) deleted the mysql-standard-pk-sync branch September 18, 2023 22:17

Duy Nguyen (nguyenaiden) mentioned this pull request Sep 18, 2023

[Source-MySQL] Enable Enhanced Standard sync with PK -> Cursor switch #30556

Merged

Conversation

Duy Nguyen (nguyenaiden) commented Sep 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Uh oh!

github-actions Bot commented Sep 7, 2023 • edited by nguyenaiden Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 11, 2023

source-snowflake test report (commit 569bfc0623) - ❌

Uh oh!

github-actions Bot commented Sep 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report for source-postgres

Uh oh!

github-actions Bot commented Sep 11, 2023

source-cockroachdb test report (commit 569bfc0623) - ❌

Uh oh!

github-actions Bot commented Sep 11, 2023

source-mysql-strict-encrypt test report (commit 569bfc0623) - ❌

Uh oh!

github-actions Bot commented Sep 11, 2023

source-oracle-strict-encrypt test report (commit 569bfc0623) - ❌

Uh oh!

github-actions Bot commented Sep 11, 2023

source-postgres test report (commit 569bfc0623) - ✅

Uh oh!

github-actions Bot commented Sep 18, 2023

source-mysql-strict-encrypt test report (commit 76c62cea72) - ❌

Uh oh!

github-actions Bot commented Sep 18, 2023

source-mysql test report (commit 76c62cea72) - ✅

Uh oh!

github-actions Bot commented Sep 18, 2023

source-postgres test report (commit 76c62cea72) - ❌

Uh oh!

github-actions Bot commented Sep 18, 2023

source-mssql test report (commit 76c62cea72) - ❌

Uh oh!

Duy Nguyen (nguyenaiden) commented Sep 18, 2023

Uh oh!

Octavia The Approver (octavia-approvington) commented Sep 18, 2023

Uh oh!

github-actions Bot commented Sep 18, 2023

source-mysql test report (commit b0c1071caf) - ❌

Uh oh!

github-actions Bot commented Sep 18, 2023

source-mssql test report (commit b0c1071caf) - ❌

Uh oh!

github-actions Bot commented Sep 18, 2023

source-postgres test report (commit b0c1071caf) - ❌

Uh oh!

github-actions Bot commented Sep 18, 2023

source-mysql-strict-encrypt test report (commit b0c1071caf) - ❌

Uh oh!

Duy Nguyen (nguyenaiden) commented Sep 7, 2023 •

edited

Loading

github-actions Bot commented Sep 7, 2023 •

edited by nguyenaiden

Loading

source-snowflake test report (commit `569bfc0623`) - ❌

github-actions Bot commented Sep 11, 2023 •

edited

Loading

source-cockroachdb test report (commit `569bfc0623`) - ❌

source-mysql-strict-encrypt test report (commit `569bfc0623`) - ❌

source-oracle-strict-encrypt test report (commit `569bfc0623`) - ❌

source-postgres test report (commit `569bfc0623`) - ✅

source-mysql-strict-encrypt test report (commit `76c62cea72`) - ❌

source-mysql test report (commit `76c62cea72`) - ✅

source-postgres test report (commit `76c62cea72`) - ❌

source-mssql test report (commit `76c62cea72`) - ❌

source-mysql test report (commit `b0c1071caf`) - ❌

source-mssql test report (commit `b0c1071caf`) - ❌

source-postgres test report (commit `b0c1071caf`) - ❌

source-mysql-strict-encrypt test report (commit `b0c1071caf`) - ❌