Enable MySQL CDC + PK 100% #30213

akashkulk · 2023-09-06T17:01:04Z

Closes #30215 and enables the MySQL CDC initial load for 100%

Fixes the integration tests, removes the legacy CDC snapshot load test

github-actions · 2023-09-06T17:01:25Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

rodireich · 2023-09-06T21:08:30Z

...sql/src/main/java/io/airbyte/integrations/source/mysql/initialsync/MySqlInitialReadUtil.java

@@ -179,6 +179,7 @@ public static InitialLoadStreams streamsForInitialPrimaryKeyLoad(final CdcStateM
                                                                   final ConfiguredAirbyteCatalog fullCatalog,
                                                                   final boolean savedOffsetStillPresentOnServer) {
    if (!savedOffsetStillPresentOnServer) {
+      LOGGER.info("FULL RESYNC DUE TO SAVED OFFSET ISSUE");


checking: intentionally left this log?

No, I need to revert this file and MySqlInitialStateIterator.java. Thanks for checking!

github-actions · 2023-09-07T02:29:36Z

source-mysql test report (commit `f4c534cc13`) - ❌

⏲️ Total pipeline duration: 319mn47s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-07T03:09:02Z

source-mysql-strict-encrypt test report (commit `f4c534cc13`) - ❌

⏲️ Total pipeline duration: 39mn15s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

github-actions · 2023-09-07T03:48:40Z

source-mysql test report (commit `c96237f61d`) - ❌

⏲️ Total pipeline duration: 37mn17s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-07T04:28:10Z

source-mysql-strict-encrypt test report (commit `c96237f61d`) - ❌

⏲️ Total pipeline duration: 39mn20s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

nguyenaiden

+1

subodh1810

Some NIT comments

MySqlCdcStateHandler class method saveStateAfterCompletionOfSnapshotOfNewStreams should now throw runtime exception cause debezium is no longer used for new tables snapshot and this method should never be triggered.
MySqlCdcProperties class method getSnapshotProperties is no longer used so should be deleted.

...yte/integrations/io/airbyte/integration_tests/sources/CdcBinlogsMySqlSourceDatatypeTest.java

akashkulk · 2023-09-07T20:31:53Z

/approve-and-merge reason="mysql tests passing locally"

octavia-approvington · 2023-09-07T20:32:27Z

Lets merge it

github-actions · 2023-09-07T23:15:05Z

source-mysql test report (commit `6fcaff50a9`) - ❌

⏲️ Total pipeline duration: 322mn26s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-07T23:55:28Z

source-mysql-strict-encrypt test report (commit `6fcaff50a9`) - ❌

⏲️ Total pipeline duration: 40mn12s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

github-actions · 2023-09-08T00:38:34Z

source-mysql-strict-encrypt test report (commit `cf3c31f0b7`) - ❌

⏲️ Total pipeline duration: 41mn17s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

github-actions · 2023-09-08T05:57:41Z

source-mysql test report (commit `cf3c31f0b7`) - ❌

⏲️ Total pipeline duration: 318mn57s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

Fix integ + unit tests. Plus gut code

5076750

octavia-squidington-iii added area/connectors Connector related issues connectors/source/mysql labels Sep 6, 2023

Bump Dockerfiles + rename tests

f2eee8f

octavia-squidington-iii added area/documentation Improvements or additions to documentation connectors/source/mysql-strict-encrypt labels Sep 6, 2023

Merge branch 'master' into 100p_rollout_mysql

f4c534c

akashkulk marked this pull request as ready for review September 6, 2023 21:07

akashkulk requested a review from a team as a code owner September 6, 2023 21:07

Merge branch 'master' into 100p_rollout_mysql

69e0307

akashkulk requested review from rodireich and prateekmukhedkar September 6, 2023 21:09

rodireich approved these changes Sep 6, 2023

View reviewed changes

akashkulk requested review from rodireich, subodh1810, a team and nguyenaiden September 6, 2023 21:09

rodireich approved these changes Sep 6, 2023

View reviewed changes

Reverting extra logging

c96237f

prateekmukhedkar approved these changes Sep 7, 2023

View reviewed changes

nguyenaiden approved these changes Sep 7, 2023

View reviewed changes

Merge branch 'master' into 100p_rollout_mysql

6fcaff5

subodh1810 approved these changes Sep 7, 2023

View reviewed changes

...yte/integrations/io/airbyte/integration_tests/sources/CdcBinlogsMySqlSourceDatatypeTest.java Outdated Show resolved Hide resolved

akashkulk added 2 commits September 7, 2023 13:06

Handling nits + deleting code

8179bbf

Revert AirbyteDebeziumHandler

e0254bb

Merge branch 'master' into 100p_rollout_mysql

f99a8ff

octavia-approvington approved these changes Sep 7, 2023

View reviewed changes

octavia-approvington merged commit 09df4f4 into master Sep 7, 2023
11 of 12 checks passed

octavia-approvington deleted the 100p_rollout_mysql branch September 7, 2023 20:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable MySQL CDC + PK 100% #30213

Enable MySQL CDC + PK 100% #30213

akashkulk commented Sep 6, 2023 •

edited

github-actions bot commented Sep 6, 2023

rodireich Sep 6, 2023

akashkulk Sep 6, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

nguyenaiden left a comment

subodh1810 left a comment

akashkulk commented Sep 7, 2023

octavia-approvington commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 8, 2023

github-actions bot commented Sep 8, 2023

Enable MySQL CDC + PK 100% #30213

Enable MySQL CDC + PK 100% #30213

Conversation

akashkulk commented Sep 6, 2023 • edited

github-actions bot commented Sep 6, 2023

Before Merging a Connector Pull Request

rodireich Sep 6, 2023

Choose a reason for hiding this comment

akashkulk Sep 6, 2023

Choose a reason for hiding this comment

github-actions bot commented Sep 7, 2023

source-mysql test report (commit f4c534cc13) - ❌

github-actions bot commented Sep 7, 2023

source-mysql-strict-encrypt test report (commit f4c534cc13) - ❌

github-actions bot commented Sep 7, 2023

source-mysql test report (commit c96237f61d) - ❌

github-actions bot commented Sep 7, 2023

source-mysql-strict-encrypt test report (commit c96237f61d) - ❌

nguyenaiden left a comment

Choose a reason for hiding this comment

subodh1810 left a comment

Choose a reason for hiding this comment

akashkulk commented Sep 7, 2023

octavia-approvington commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

source-mysql test report (commit 6fcaff50a9) - ❌

github-actions bot commented Sep 7, 2023

source-mysql-strict-encrypt test report (commit 6fcaff50a9) - ❌

github-actions bot commented Sep 8, 2023

source-mysql-strict-encrypt test report (commit cf3c31f0b7) - ❌

github-actions bot commented Sep 8, 2023

source-mysql test report (commit cf3c31f0b7) - ❌

akashkulk commented Sep 6, 2023 •

edited

source-mysql test report (commit `f4c534cc13`) - ❌

source-mysql-strict-encrypt test report (commit `f4c534cc13`) - ❌

source-mysql test report (commit `c96237f61d`) - ❌

source-mysql-strict-encrypt test report (commit `c96237f61d`) - ❌

source-mysql test report (commit `6fcaff50a9`) - ❌

source-mysql-strict-encrypt test report (commit `6fcaff50a9`) - ❌

source-mysql-strict-encrypt test report (commit `cf3c31f0b7`) - ❌

source-mysql test report (commit `cf3c31f0b7`) - ❌