🐛 Only take table size info for streams syncing via primary key by rodireich · Pull Request #30375 · airbytehq/airbyte

Rodi Reich Zilberman (rodireich) · 2023-09-13T04:16:00Z

What

If at least one stream is syncing via primary key, we attempt to get table info size for all,
including full refresh streams and views.
Because views don't have physical byte size we run into an error.
Closes #30363.

How

Only take table info size for the streams that need it, namely primary key syncing streams.

github-actions · 2023-09-13T04:16:19Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2023-09-13T09:38:57Z

source-mysql test report (commit `7adcec95dc`) - ❌

⏲️ Total pipeline duration: 321mn19s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-13T15:04:18Z

source-mysql test report (commit `f399d58161`) - ❌

⏲️ Total pipeline duration: 322mn53s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-13T15:44:37Z

source-mysql-strict-encrypt test report (commit `f399d58161`) - ❌

⏲️ Total pipeline duration: 40mn08s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

Subodh Chaturvedi (subodh1810)

We should just swallow any exception in the method getTableSizeInfoForStreams in class MySqlQueryUtils. We are already swallowing SQLException.

Also add a null check in that method for values before trying to parse them to long

Rodi Reich Zilberman (rodireich) · 2023-09-13T16:11:21Z

I see that in case there is no table size info, MySqlInitialLoadHandler would fall back to a default chunk size of 1m rows

    if (tableSizeInfo == null || tableSizeInfo.tableSize() == 0 || tableSizeInfo.avgRowLength() == 0) {
      LOGGER.info("Chunk size could not be determined for pair: {}, defaulting to {} rows", pair, DEFAULT_CHUNK_SIZE);
      return DEFAULT_CHUNK_SIZE;
    }

I'll add the suggested change so we never fail a sync due to table size failure

Rodi Reich Zilberman (rodireich) · 2023-09-13T17:15:40Z

/legacy-test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/6175707390
❌ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/6175707390
🐛 https://gradle.com/s/ihgfidlelrf5w

Build Failed

Test summary info:

Could not find result summary

github-actions · 2023-09-13T17:35:05Z

source-mysql-strict-encrypt test report (commit `3aa0010287`) - ❌

⏲️ Total pipeline duration: 42mn21s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

…ce-mysql-error-when-reading-table-size-causing-syncs-to-fail

Rodi Reich Zilberman (rodireich) · 2023-09-13T17:50:28Z

All tests passing locally

…e-size-causing-syncs-to-fail

Prateek Mukhedkar (prateekmukhedkar) · 2023-09-13T18:48:57Z

/legacy-test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/6176609667
❌ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/6176609667
🐛 https://gradle.com/s/45tytjy5zv43c

Build Failed

Test summary info:

Could not find result summary

Subodh Chaturvedi (subodh1810)

I pushed a commit as discussed over call. Please make sure to the tests are green locally (CI is broken) before merging.

Rodi Reich Zilberman (rodireich) · 2023-09-13T19:34:30Z

/approve-and-merge reason="Tests pass for MySQL locally, CI is broken for mysql"

Octavia The Approver (octavia-approvington) · 2023-09-13T19:35:07Z

Lets merge it

github-actions · 2023-09-13T22:54:48Z

source-mysql test report (commit `3aa0010287`) - ❌

⏲️ Total pipeline duration: 319mn32s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-14T04:18:55Z

source-mysql test report (commit `c603e18c66`) - ❌

⏲️ Total pipeline duration: 322mn19s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-14T04:58:33Z

source-mysql-strict-encrypt test report (commit `c603e18c66`) - ❌

⏲️ Total pipeline duration: 39mn27s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

Only take table size info for streams syncing via primary key

7adcec9

Rodi Reich Zilberman (rodireich) requested a review from Prateek Mukhedkar (prateekmukhedkar) September 13, 2023 04:16

Rodi Reich Zilberman (rodireich) requested a review from a team as a code owner September 13, 2023 04:16

Rodi Reich Zilberman (rodireich) linked an issue Sep 13, 2023 that may be closed by this pull request

[source-mysql] Error when reading table size causing syncs to fail #30363

Closed

1 task

Octavia Squidington III (octavia-squidington-iii) added the area/connectors Connector related issues label Sep 13, 2023

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/mysql label Sep 13, 2023

Rodi Reich Zilberman (rodireich) changed the title ~~Only take table size info for streams syncing via primary key~~ 🐛 Only take table size info for streams syncing via primary key Sep 13, 2023

bump versions update changelog

f399d58

Octavia Squidington III (octavia-squidington-iii) added area/documentation Improvements or additions to documentation connectors/source/mysql-strict-encrypt labels Sep 13, 2023

Subodh Chaturvedi (subodh1810) requested changes Sep 13, 2023

View reviewed changes

Keep going in case table size estimate fails

3aa0010

Rodi Reich Zilberman (rodireich) requested a review from Subodh Chaturvedi (subodh1810) September 13, 2023 16:51

Rodi Reich Zilberman (rodireich) added 2 commits September 13, 2023 10:46

Merge branch 'master' of github.com:airbytehq/airbyte into 30363-sour…

6999eb6

…ce-mysql-error-when-reading-table-size-causing-syncs-to-fail

bump versions

b78790f

Rodi Reich Zilberman (rodireich) added 2 commits September 13, 2023 11:42

Merge branch 'master' into 30363-source-mysql-error-when-reading-tabl…

edbbd21

…e-size-causing-syncs-to-fail

bump version

02e5d30

Prateek Mukhedkar (prateekmukhedkar) approved these changes Sep 13, 2023

View reviewed changes

swallow exception in method getTableEstimate

e7ee2ab

Subodh Chaturvedi (subodh1810) approved these changes Sep 13, 2023

View reviewed changes

Octavia The Approver (octavia-approvington) approved these changes Sep 13, 2023

View reviewed changes

Octavia The Approver (octavia-approvington) merged commit 0936143 into master Sep 13, 2023

Octavia The Approver (octavia-approvington) deleted the 30363-source-mysql-error-when-reading-table-size-causing-syncs-to-fail branch September 13, 2023 19:35

Uh oh!

Conversation

Rodi Reich Zilberman (rodireich) commented Sep 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Uh oh!

github-actions Bot commented Sep 13, 2023 • edited by rodireich Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Uh oh!

github-actions Bot commented Sep 13, 2023

source-mysql test report (commit 7adcec95dc) - ❌

Uh oh!

github-actions Bot commented Sep 13, 2023

source-mysql test report (commit f399d58161) - ❌

Uh oh!

github-actions Bot commented Sep 13, 2023

source-mysql-strict-encrypt test report (commit f399d58161) - ❌

Uh oh!

Subodh Chaturvedi (subodh1810) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rodi Reich Zilberman (rodireich) commented Sep 13, 2023

Uh oh!

Rodi Reich Zilberman (rodireich) commented Sep 13, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

github-actions Bot commented Sep 13, 2023

source-mysql-strict-encrypt test report (commit 3aa0010287) - ❌

Uh oh!

Rodi Reich Zilberman (rodireich) commented Sep 13, 2023

Uh oh!

Prateek Mukhedkar (prateekmukhedkar) commented Sep 13, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

Subodh Chaturvedi (subodh1810) left a comment

Choose a reason for hiding this comment

Uh oh!

Rodi Reich Zilberman (rodireich) commented Sep 13, 2023

Uh oh!

Octavia The Approver (octavia-approvington) commented Sep 13, 2023

Uh oh!

github-actions Bot commented Sep 13, 2023

source-mysql test report (commit 3aa0010287) - ❌

Uh oh!

github-actions Bot commented Sep 14, 2023

source-mysql test report (commit c603e18c66) - ❌

Uh oh!

github-actions Bot commented Sep 14, 2023

source-mysql-strict-encrypt test report (commit c603e18c66) - ❌

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Rodi Reich Zilberman (rodireich) commented Sep 13, 2023 •

edited

Loading

github-actions Bot commented Sep 13, 2023 •

edited by rodireich

Loading

source-mysql test report (commit `7adcec95dc`) - ❌

source-mysql test report (commit `f399d58161`) - ❌

source-mysql-strict-encrypt test report (commit `f399d58161`) - ❌

Subodh Chaturvedi (subodh1810) left a comment •

edited

Loading

Rodi Reich Zilberman (rodireich) commented Sep 13, 2023 •

edited by github-actions Bot

Loading

source-mysql-strict-encrypt test report (commit `3aa0010287`) - ❌

Prateek Mukhedkar (prateekmukhedkar) commented Sep 13, 2023 •

edited by github-actions Bot

Loading

source-mysql test report (commit `3aa0010287`) - ❌

source-mysql test report (commit `c603e18c66`) - ❌

source-mysql-strict-encrypt test report (commit `c603e18c66`) - ❌