[Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over #30270

nguyenaiden · 2023-09-07T23:22:49Z

What

Utilize Primary keys for initial sync then switch over to user defined cursor once high water mark is found and persisted in state.

How

With normal incremental syncs now, we will be using Primary key for the initial sync and retrieve the high water mark in the beginning and construct all of the relevant Primary Key and Standard Sync iterators at the beginning of the sync.

This includes a SELECT MAX query to retrieve max cursor value and populate the initial and final StreamState to provide a checkpoint so that there won't be any missing records on subsequent syncs.

Note: The getIncrementalIterators method in the MysqlInitialLoadHandler class now has logic to handle the edge case of when the customer de-select a primary key column. Solution here is to re-add the column names to the selectedDatabaseFields in order to construct the correct SELECT query, but those columns will not be part of the record as the CatalogHelpers.getTopLevelFieldNames(airbyteStream) will not include the de-selected fields.

Tests with flag turned on is in the file MySqlPkJdbcSourceAcceptanceTest.java

github-actions · 2023-09-07T23:23:16Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

akashkulk · 2023-09-08T16:25:23Z

airbyte-cdk/python/airbyte_cdk/sources/declarative/models/declarative_component_schema.py

Why is a change to this file needed?

.../connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java

akashkulk · 2023-09-08T16:36:10Z

...nectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlQueryUtils.java

+        LOGGER.info("Querying max cursor value for {}.{}", namespace, name);
+        final String cursorField = cursorInfoOptional.get().getCursorField();
+        final String cursorBasedSyncStatusQuery = String.format(MAX_CURSOR_VALUE_QUERY,
+            cursorField,


We should escape quote cursor fields like in this PR : https://github.com/airbytehq/airbyte/pull/30059/files

akashkulk · 2023-09-08T16:40:28Z

...ava/io/airbyte/integrations/source/mysql/initialsync/MySqlInitialLoadGlobalStateManager.java

    this.cdcState = cdcState;
    this.pairToPrimaryKeyLoadStatus = initPairToPrimaryKeyLoadStatusMap(initialLoadStreams.pairToInitialLoadStatus());
    this.pairToPrimaryKeyInfo = pairToPrimaryKeyInfo;
    this.streamsThatHaveCompletedSnapshot = initStreamsCompletedSnapshot(initialLoadStreams, catalog);
  }

-  private static Set<AirbyteStreamNameNamespacePair> initStreamsCompletedSnapshot(final InitialLoadStreams initialLoadStreams,


Why are we deleting this code?

I moved this to the Utils class so I can use it as well

can we move this back now that this is only being used in here! Just trying to minimise the changes.

I'm good with that! On it

akashkulk · 2023-09-08T16:43:44Z

.../connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java

@@ -338,6 +349,54 @@ public List<AutoCloseableIterator<AirbyteMessage>> getIncrementalIterators(final
      LOGGER.info("Using PK + CDC");
      return MySqlInitialReadUtil.getCdcReadIterators(database, catalog, tableNameToTable, stateManager, emittedAt, getQuoteString());
    } else {
+      if (isAnyStreamIncrementalSyncMode(catalog)) {
+        final MySqlCursorBasedStateManager cursorBasedStateManager = new MySqlCursorBasedStateManager(stateManager.getRawStateMessages(), catalog);


Can we move this to MySqlInitialReadUtil? I assume this is the code that is doing the main logic of figuring out which streams to sync via pk?

Moved it! I actually eliminated the filtering based on state_type = cursor_based since anything that's not PK is implicitly cursor_based.

akashkulk · 2023-09-08T16:57:49Z

...ain/java/io/airbyte/integrations/source/mysql/cursor_based/MySqlCursorBasedStateManager.java

+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class MySqlCursorBasedStateManager extends StreamStateManager {


This is similar to the PostgresCursorBasedStateManager class correct? Basically just to enable writing the state_type and version keys? Can we add a comment for that

IIRC we had a ticket to refactor this in StreamStateManager when we implemented source-mysql - do you think we should do that here?

akashkulk · 2023-09-08T17:28:12Z

...l/src/main/java/io/airbyte/integrations/source/mysql/cursor_based/MysqlCursorBasedUtils.java

+
+  }
+
+  public record StreamsCategorised(InitialLoadStreams initialLoadStreams,


can we remove this and just keep InitialLoadStreams? Anything that is not in InitialLoadStreams would implicitly be a cursor based streams right?

Yep! Addressed above.

github-actions · 2023-09-11T05:00:24Z

source-snowflake test report (commit `569bfc0623`) - ❌

⏲️ Total pipeline duration: 13mn02s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-snowflake/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-snowflake test

github-actions · 2023-09-11T05:02:42Z

Coverage report for source-postgres

File	Coverage [87.89%]	🍏
PostgresSource.java	88.62%	🍏
PostgresUtils.java	84.9%	🍏

Total Project Coverage	70.56%	🍏

github-actions · 2023-09-11T15:03:11Z

source-cockroachdb test report (commit `569bfc0623`) - ❌

⏲️ Total pipeline duration: 602mn36s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-cockroachdb docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-cockroachdb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-cockroachdb test

github-actions · 2023-09-11T15:44:43Z

source-mysql-strict-encrypt test report (commit `569bfc0623`) - ❌

⏲️ Total pipeline duration: 41mn21s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

github-actions · 2023-09-11T16:06:26Z

source-oracle-strict-encrypt test report (commit `569bfc0623`) - ❌

⏲️ Total pipeline duration: 21mn31s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-oracle-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-oracle-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-oracle-strict-encrypt test

github-actions · 2023-09-11T16:36:03Z

source-postgres test report (commit `569bfc0623`) - ✅

⏲️ Total pipeline duration: 29mn26s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-09-18T21:55:37Z

source-mysql test report (commit `76c62cea72`) - ✅

⏲️ Total pipeline duration: 02mn36s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-18T21:58:28Z

source-postgres test report (commit `76c62cea72`) - ❌

⏲️ Total pipeline duration: 02mn31s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-09-18T21:58:55Z

source-mssql test report (commit `76c62cea72`) - ❌

⏲️ Total pipeline duration: 13.21s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql test

WIP WIP Functional feature done

Formatting

nguyenaiden · 2023-09-18T22:16:18Z

/approve-and-merge reason="ci passes and features are gated behind flag". Source-postgres and mssql changes are non-functional. "

octavia-approvington · 2023-09-18T22:16:58Z

Jerry would be proud

github-actions · 2023-09-18T22:43:20Z

source-mysql test report (commit `b0c1071caf`) - ❌

⏲️ Total pipeline duration: 30mn51s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-18T22:43:50Z

source-mssql test report (commit `b0c1071caf`) - ❌

⏲️ Total pipeline duration: 6.00s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql test

github-actions · 2023-09-18T22:46:26Z

source-postgres test report (commit `b0c1071caf`) - ❌

⏲️ Total pipeline duration: 01mn51s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-09-18T22:55:26Z

source-mysql-strict-encrypt test report (commit `b0c1071caf`) - ❌

⏲️ Total pipeline duration: 08mn25s

Step	Result
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/source/mysql connectors/source/jdbc labels Sep 7, 2023

akashkulk reviewed Sep 8, 2023

View reviewed changes

.../connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java Show resolved Hide resolved

akashkulk reviewed Sep 8, 2023

View reviewed changes

octavia-squidington-iii added connectors/source/cockroachdb connectors/source/oracle connectors/source/postgres connectors/source/snowflake connectors/source/mysql-strict-encrypt connectors/source/oracle-strict-encrypt labels Sep 11, 2023

nguyenaiden requested a review from subodh1810 September 11, 2023 04:44

nguyenaiden marked this pull request as ready for review September 11, 2023 04:45

nguyenaiden requested review from a team as code owners September 11, 2023 04:45

nguyenaiden changed the title ~~[Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over [WIP]~~ [Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over Sep 11, 2023

nguyenaiden requested a review from rodireich September 11, 2023 04:45

nguyenaiden added 14 commits September 18, 2023 15:06

WIP standard sync

532bc9a

WIP WIP Functional feature done

uncomment test

6b7d05b

Separated out cursor and pk streams

0527fa1

Feature complete

bb67fb0

Added necessary override decorators and removed unneeded null checks

f7583ae

Addressed comments

39f43dd

Added override back to Oracle test

aa17ddb

Localize changes to just MysqlPkJdbcSourceAcceptanceTests

58a6e39

Formatting

Reverted changes to cockroachdb test

e0bbf87

Bumped metadata and update changelog

8dfdd4c

Update getStateData override to not use legacy states

4eedbf9

Bumped dockerfile

3c31b58

Added pk test for strict encrypt as well

4a3df49

Bump mysql version after flag has been removed

b0c1071

nguyenaiden force-pushed the mysql-standard-pk-sync branch from 372e55d to b0c1071 Compare September 18, 2023 22:07

octavia-squidington-iii removed the area/documentation Improvements or additions to documentation label Sep 18, 2023

octavia-approvington approved these changes Sep 18, 2023

View reviewed changes

octavia-approvington merged commit f226503 into master Sep 18, 2023
19 of 26 checks passed

octavia-approvington deleted the mysql-standard-pk-sync branch September 18, 2023 22:17

nguyenaiden mentioned this pull request Sep 18, 2023

[Source-MySQL] Enable Enhanced Standard sync with PK -> Cursor switch #30556

Merged

dalebradman mentioned this pull request Sep 20, 2023

✨ Adding images and videos streams to TikTok Marketing connector tryarcane/airbyte#1

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over #30270

[Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over #30270

nguyenaiden commented Sep 7, 2023 •

edited

github-actions bot commented Sep 7, 2023 •

edited by nguyenaiden

akashkulk Sep 8, 2023

akashkulk Sep 8, 2023

akashkulk Sep 8, 2023

nguyenaiden Sep 11, 2023

subodh1810 Sep 13, 2023

nguyenaiden Sep 13, 2023

akashkulk Sep 8, 2023

nguyenaiden Sep 11, 2023

akashkulk Sep 8, 2023

akashkulk Sep 8, 2023

nguyenaiden Sep 11, 2023

github-actions bot commented Sep 11, 2023

github-actions bot commented Sep 11, 2023 •

edited

github-actions bot commented Sep 11, 2023

github-actions bot commented Sep 11, 2023

github-actions bot commented Sep 11, 2023

github-actions bot commented Sep 11, 2023

github-actions bot commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

nguyenaiden commented Sep 18, 2023

octavia-approvington commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

github-actions bot commented Sep 18, 2023


		}

		public record StreamsCategorised(InitialLoadStreams initialLoadStreams,

[Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over #30270

[Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over #30270

Conversation

nguyenaiden commented Sep 7, 2023 • edited

What

How

github-actions bot commented Sep 7, 2023 • edited by nguyenaiden

Before Merging a Connector Pull Request

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 11, 2023

source-snowflake test report (commit 569bfc0623) - ❌

github-actions bot commented Sep 11, 2023 • edited

Coverage report for source-postgres

github-actions bot commented Sep 11, 2023

source-cockroachdb test report (commit 569bfc0623) - ❌

github-actions bot commented Sep 11, 2023

source-mysql-strict-encrypt test report (commit 569bfc0623) - ❌

github-actions bot commented Sep 11, 2023

source-oracle-strict-encrypt test report (commit 569bfc0623) - ❌

github-actions bot commented Sep 11, 2023

source-postgres test report (commit 569bfc0623) - ✅

github-actions bot commented Sep 18, 2023

source-mysql test report (commit 76c62cea72) - ✅

github-actions bot commented Sep 18, 2023

source-postgres test report (commit 76c62cea72) - ❌

github-actions bot commented Sep 18, 2023

source-mssql test report (commit 76c62cea72) - ❌

nguyenaiden commented Sep 18, 2023

octavia-approvington commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

source-mysql test report (commit b0c1071caf) - ❌

github-actions bot commented Sep 18, 2023

source-mssql test report (commit b0c1071caf) - ❌

github-actions bot commented Sep 18, 2023

source-postgres test report (commit b0c1071caf) - ❌

github-actions bot commented Sep 18, 2023

source-mysql-strict-encrypt test report (commit b0c1071caf) - ❌

nguyenaiden commented Sep 7, 2023 •

edited

github-actions bot commented Sep 7, 2023 •

edited by nguyenaiden

source-snowflake test report (commit `569bfc0623`) - ❌

github-actions bot commented Sep 11, 2023 •

edited

source-cockroachdb test report (commit `569bfc0623`) - ❌

source-mysql-strict-encrypt test report (commit `569bfc0623`) - ❌

source-oracle-strict-encrypt test report (commit `569bfc0623`) - ❌

source-postgres test report (commit `569bfc0623`) - ✅

source-mysql test report (commit `76c62cea72`) - ✅

source-postgres test report (commit `76c62cea72`) - ❌

source-mssql test report (commit `76c62cea72`) - ❌

source-mysql test report (commit `b0c1071caf`) - ❌

source-mssql test report (commit `b0c1071caf`) - ❌

source-postgres test report (commit `b0c1071caf`) - ❌

source-mysql-strict-encrypt test report (commit `b0c1071caf`) - ❌