Update MySQL streaming config by akashkulk · Pull Request #30744 · airbytehq/airbyte

Akash Kulkarni (akashkulk) · 2023-09-25T22:00:07Z

Fixes issue described https://airbytehq-team.slack.com/archives/C043JHEEYKG/p1695336111738279

Some notes :

The auto-commit behavior was added to primarily handle the Postgres use case of respecting fetch size : https://jdbc.postgresql.org/documentation/query/
MySql extends the default behavior due to lack of support for fetch size. This is hanled in MySqlStreamingQueryConfig.java
Step 2 was necessary due to poor interactions of using cursor fetch on MySQL DB. Fetch size is set to Integer.MIN_VALUE to disable client side caching and the the result set arrives as soon as the responses are ready. See 🎉 8890 Source MySql: Fix large table issue by fetch size #17236 for more context.
Seperately, there seems to be an issue in the JDBC driver handling of resetting auto-commit values for MariaDB. Unsure what the root cause is there, but the code no longer crashes when auto-commit values don't need to be changed.
The auto-commit setting is not needed for MySQL, but was copied over from the subclass that primarily handles the Postgres connection configs. Hence, this should be safe to remove.

vercel · 2023-09-25T22:00:13Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Sep 26, 2023 10:09pm

github-actions · 2023-09-25T22:00:29Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

airbyte-oss-build-runner · 2023-09-26T00:51:01Z

source-mysql test report (commit `80d5c2fc00`) - ✅

⏲️ Total pipeline duration: 27mn49s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

airbyte-oss-build-runner · 2023-09-26T00:58:54Z

source-mysql-strict-encrypt test report (commit `80d5c2fc00`) - ✅

⏲️ Total pipeline duration: 07mn43s

Step	Result
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

Rodi Reich Zilberman (rodireich) · 2023-09-26T16:24:25Z

            try {
-              connection.setAutoCommit(true);
+              if (!connection.getAutoCommit()) {
+                connection.setAutoCommit(true);


I not familiar with the intricacies here
But Is this logically any different ?
in both before and after we end with autoCommit == true

Logically it could be different if we had multiple transactions within a single streaming query which mutate the source DB (which we don't)

The setting autocommit = true was just done to deal with a quirk in the POstgres JDBC driver + fetch size interaction which does not apply for mysql.

Akash Kulkarni (akashkulk) · 2023-09-26T19:46:01Z

/connector-performance connector=connectors/source-mysql

Note: The following `dataset=` values are supported: `1m`_(default), `10m`, `20m`,

bottleneck_stream1, bottleneck_stream_randomseed. For destinations only: you can also use stream-numbers=Nto simulate N number of parallel streams. Additionally,sync-mode=incrementalis supported for destinations. For example:dataset=1m stream-numbers=2 sync-mode=incremental`

🏃 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/6317576894.

airbyte-oss-build-runner · 2023-09-26T19:50:16Z

source-mysql-strict-encrypt test report (commit `22860626de`) - ✅

⏲️ Total pipeline duration: 03mn09s

Step	Result
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

Akash Kulkarni (akashkulk) · 2023-09-26T19:51:03Z

/connector-performance connector=connectors/source-mysql dataset=20m

Note: The following `dataset=` values are supported: `1m`_(default), `10m`, `20m`,

bottleneck_stream1, bottleneck_stream_randomseed. For destinations only: you can also use stream-numbers=Nto simulate N number of parallel streams. Additionally,sync-mode=incrementalis supported for destinations. For example:dataset=1m stream-numbers=2 sync-mode=incremental`

🏃 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/6317621188.

airbyte-oss-build-runner · 2023-09-26T19:52:44Z

source-mysql test report (commit `22860626de`) - ✅

⏲️ Total pipeline duration: 02mn18s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

airbyte-oss-build-runner · 2023-09-26T22:15:06Z

source-mysql-strict-encrypt test report (commit `434e39b0ee`) - ✅

⏲️ Total pipeline duration: 03mn22s

Step	Result
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

airbyte-oss-build-runner · 2023-09-26T22:17:35Z

source-mysql test report (commit `434e39b0ee`) - ✅

⏲️ Total pipeline duration: 02mn19s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

Update MySQL streaming config

8d735d5

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues connectors/source/mysql labels Sep 25, 2023

Update metadata

309f237

Octavia Squidington III (octavia-squidington-iii) added area/documentation Improvements or additions to documentation connectors/source/mysql-strict-encrypt labels Sep 26, 2023

Update docs

06a8dcd

vercel Bot deployed to Preview September 26, 2023 00:05

View deployment

Merge branch 'master' into debug_mysql_indexOOB

80d5c2f

Akash Kulkarni (akashkulk) marked this pull request as ready for review September 26, 2023 00:20

Akash Kulkarni (akashkulk) requested a review from a team as a code owner September 26, 2023 00:20

Akash Kulkarni (akashkulk) requested review from Duy Nguyen (nguyenaiden), Prateek Mukhedkar (prateekmukhedkar), Rodi Reich Zilberman (rodireich) and Subodh Chaturvedi (subodh1810) September 26, 2023 00:21

vercel Bot deployed to Preview September 26, 2023 00:23

View deployment

Rodi Reich Zilberman (rodireich) reviewed Sep 26, 2023

View reviewed changes

Subodh Chaturvedi (subodh1810) approved these changes Sep 26, 2023

View reviewed changes

Akash Kulkarni (akashkulk) requested a review from Marius Posta (postamar) September 26, 2023 16:45

Merge branch 'master' into debug_mysql_indexOOB

2286062

vercel Bot deployed to Preview September 26, 2023 19:46

View deployment

Rodi Reich Zilberman (rodireich) approved these changes Sep 26, 2023

View reviewed changes

Merge branch 'master' into debug_mysql_indexOOB

434e39b

Akash Kulkarni (akashkulk) merged commit 2097c0e into master Sep 26, 2023

Akash Kulkarni (akashkulk) deleted the debug_mysql_indexOOB branch September 26, 2023 22:45

Navigation Menu

Repository navigation

Update MySQL streaming config - #30744

Update MySQL streaming config#30744
Akash Kulkarni (akashkulk) merged 6 commits into
masterfrom
debug_mysql_indexOOB

Akash Kulkarni (akashkulk) commented Sep 25, 2023 •

edited

Loading

Uh oh!

vercel Bot commented Sep 25, 2023 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 25, 2023

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

Uh oh!

Rodi Reich Zilberman (rodireich) Sep 26, 2023

Uh oh!

Akash Kulkarni (akashkulk) Sep 26, 2023

Uh oh!

Akash Kulkarni (akashkulk) commented Sep 26, 2023 •

edited by github-actions Bot

Loading

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

Uh oh!

Akash Kulkarni (akashkulk) commented Sep 26, 2023 •

edited by github-actions Bot

Loading

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Akash Kulkarni (akashkulk) commented Sep 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel Bot commented Sep 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 25, 2023

Before Merging a Connector Pull Request

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

source-mysql test report (commit 80d5c2fc00) - ✅

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

source-mysql-strict-encrypt test report (commit 80d5c2fc00) - ✅

Uh oh!

Rodi Reich Zilberman (rodireich) Sep 26, 2023

Choose a reason for hiding this comment

Uh oh!

Akash Kulkarni (akashkulk) Sep 26, 2023

Choose a reason for hiding this comment

Uh oh!

Akash Kulkarni (akashkulk) commented Sep 26, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Note: The following dataset= values are supported: 1m(default), 10m, 20m,

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

source-mysql-strict-encrypt test report (commit 22860626de) - ✅

Uh oh!

Akash Kulkarni (akashkulk) commented Sep 26, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Note: The following dataset= values are supported: 1m(default), 10m, 20m,

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

source-mysql test report (commit 22860626de) - ✅

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

source-mysql-strict-encrypt test report (commit 434e39b0ee) - ✅

Uh oh!

airbyte-oss-build-runner commented Sep 26, 2023

source-mysql test report (commit 434e39b0ee) - ✅

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Akash Kulkarni (akashkulk) commented Sep 25, 2023 •

edited

Loading

vercel Bot commented Sep 25, 2023 •

edited

Loading

source-mysql test report (commit `80d5c2fc00`) - ✅

source-mysql-strict-encrypt test report (commit `80d5c2fc00`) - ✅

Akash Kulkarni (akashkulk) commented Sep 26, 2023 •

edited by github-actions Bot

Loading

Note: The following `dataset=` values are supported: `1m`_(default), `10m`, `20m`,

source-mysql-strict-encrypt test report (commit `22860626de`) - ✅

Akash Kulkarni (akashkulk) commented Sep 26, 2023 •

edited by github-actions Bot

Loading

Note: The following `dataset=` values are supported: `1m`_(default), `10m`, `20m`,

source-mysql test report (commit `22860626de`) - ✅

source-mysql-strict-encrypt test report (commit `434e39b0ee`) - ✅

source-mysql test report (commit `434e39b0ee`) - ✅