[Source-MySQL] Correctly set Debezium MySQL connector database timezone property by nguyenaiden · Pull Request #30333 · airbytehq/airbyte

Duy Nguyen (nguyenaiden) · 2023-09-12T02:30:16Z

What

I was able to reproduce it via running a local MySQL instance inside a Docker container with the specifically problematic CEST timezone and ran a sync on OSS.

How

The database timezone parameter was incorrectly set to database.serverTimezone, but per Debezium docs, the correct property is database.connectionTimezone.

github-actions · 2023-09-12T02:30:39Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2023-09-12T07:55:59Z

source-mysql test report (commit `ff440b155c`) - ❌

⏲️ Total pipeline duration: 322mn09s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

Subodh Chaturvedi (subodh1810)

Interesting! I bet when we added that property it was called serverTimezone but they changed it in one of the upgrades! Can you add a test for this.

Subodh Chaturvedi (subodh1810) · 2023-09-12T08:39:04Z

Can you add a link to the exact section in Debezium docs where this property is listed

https://debezium.io/documentation/reference/stable/connectors/mysql.html
The time zone will be queried from the server by default. If this fails, it must be specified explicitly by the database connectionTimeZone MySQL configuration option. For example, if the database’s time zone (either globally or configured for the connector by means of the connectionTimeZone option) is "America/Los_Angeles", the TIMESTAMP value "2018-06-20 06:37:03" is represented by a ZonedTimestamp with the value "2018-06-20T13:37:03Z".

If this fails
Do we know for a fact there was a failure?

Is the failure here because the server timezone is not IANA compliant?
That is CEST rather than Europe/Berlin

github-actions · 2023-09-12T08:41:38Z

source-mysql-strict-encrypt test report (commit `b90bd686bd`) - ❌

⏲️ Total pipeline duration: 43mn19s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

github-actions · 2023-09-12T14:01:14Z

source-mysql test report (commit `b90bd686bd`) - ❌

⏲️ Total pipeline duration: 319mn25s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

…e property

github-actions · 2023-09-12T22:40:16Z

source-mysql-strict-encrypt test report (commit `b9c1649a9d`) - ❌

⏲️ Total pipeline duration: 43mn02s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

github-actions · 2023-09-13T04:00:07Z

source-mysql test report (commit `b9c1649a9d`) - ❌

⏲️ Total pipeline duration: 319mn41s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-13T09:23:39Z

source-mysql test report (commit `c93a123e52`) - ❌

⏲️ Total pipeline duration: 321mn42s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-13T10:03:52Z

source-mysql-strict-encrypt test report (commit `c93a123e52`) - ❌

⏲️ Total pipeline duration: 40mn02s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

Prateek Mukhedkar (prateekmukhedkar) · 2023-09-13T20:25:14Z

I applied these changes and was able to sync a MariaDB hosted MySQL database in CDC mode, which was failing with this error otherwise.
Caused by: com.mysql.cj.exceptions.InvalidConnectionAttributeException: The server time zone value 'unknown' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the 'connectionTimeZone' configuration property) to use a more specific time zone value if you want to utilize time zone support.

# Conflicts: # airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml # airbyte-integrations/connectors/source-mysql/metadata.yaml # airbyte-integrations/connectors/source-mysql/src/test/java/io/airbyte/integrations/source/mysql/CdcMysqlSourceTest.java # docs/integrations/sources/mysql.md

Subodh Chaturvedi (subodh1810)

I pushed a commit to bring in the latest changes from master and resolve conflicts + add more assertions to the test (around the sync completing and asserting data).

Please run all the tests locally before merging as CI is broken.

github-actions · 2023-09-14T14:12:06Z

source-mysql test report (commit `0eb24255d0`) - ❌

⏲️ Total pipeline duration: 321mn45s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-14T14:48:52Z

source-mysql-strict-encrypt test report (commit `0eb24255d0`) - ❌

⏲️ Total pipeline duration: 36mn35s

Step	Result
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Unit Tests	❌
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

Duy Nguyen (nguyenaiden) · 2023-09-14T21:09:38Z

/approve-and-merge reason="CI is down and all expected tests passed locally"

Octavia The Approver (octavia-approvington) · 2023-09-14T21:10:12Z

Enough! Let's do this

github-actions · 2023-09-14T22:31:05Z

source-mysql test report (commit `8400b75bac`) - ❌

⏲️ Total pipeline duration: 321mn44s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-09-14T23:07:52Z

source-mysql-strict-encrypt test report (commit `8400b75bac`) - ❌

⏲️ Total pipeline duration: 36mn37s

Step	Result
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Unit Tests	❌
Java Connector Integration Tests	❌
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues connectors/source/mysql labels Sep 12, 2023

Duy Nguyen (nguyenaiden) marked this pull request as ready for review September 12, 2023 02:31

Duy Nguyen (nguyenaiden) requested a review from a team as a code owner September 12, 2023 02:31

Duy Nguyen (nguyenaiden) force-pushed the mysql-cdc-invalid-timezone branch from ff440b1 to b90bd68 Compare September 12, 2023 02:34

Octavia Squidington III (octavia-squidington-iii) added area/documentation Improvements or additions to documentation connectors/source/mysql-strict-encrypt labels Sep 12, 2023

Duy Nguyen (nguyenaiden) requested review from Prateek Mukhedkar (prateekmukhedkar), Rodi Reich Zilberman (rodireich) and Subodh Chaturvedi (subodh1810) September 12, 2023 02:34

Subodh Chaturvedi (subodh1810) reviewed Sep 12, 2023

View reviewed changes

nguyenaiden added 3 commits September 12, 2023 14:55

Update mysql debezium serverTimezone to the correct connectionTimezon…

5168292

…e property

Bumped metadata versions

e7d0ad6

Added test to verify connectionTimeZone parameter passthrough

570bdaf

Duy Nguyen (nguyenaiden) requested a review from Subodh Chaturvedi (subodh1810) September 12, 2023 21:55

Duy Nguyen (nguyenaiden) force-pushed the mysql-cdc-invalid-timezone branch from b9c1649 to 570bdaf Compare September 12, 2023 21:55

Added comments for docs

c222b4a

Clean up test and added links to docs and mysql conf key

c93a123

Prateek Mukhedkar (prateekmukhedkar) approved these changes Sep 13, 2023

View reviewed changes

Subodh Chaturvedi (subodh1810) added 2 commits September 14, 2023 13:52

resolve conflicts with master + update test to run sync and assert data

0eb2425

Subodh Chaturvedi (subodh1810) approved these changes Sep 14, 2023

View reviewed changes

Merge branch 'master' into mysql-cdc-invalid-timezone

8400b75

Octavia The Approver (octavia-approvington) approved these changes Sep 14, 2023

View reviewed changes

Octavia The Approver (octavia-approvington) merged commit fca0b6f into master Sep 14, 2023

Octavia The Approver (octavia-approvington) deleted the mysql-cdc-invalid-timezone branch September 14, 2023 21:10

Conversation

Duy Nguyen (nguyenaiden) commented Sep 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Uh oh!

github-actions Bot commented Sep 12, 2023 • edited by nguyenaiden Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Uh oh!

github-actions Bot commented Sep 12, 2023

source-mysql test report (commit ff440b155c) - ❌

Uh oh!

Subodh Chaturvedi (subodh1810) left a comment

Choose a reason for hiding this comment

Uh oh!

Subodh Chaturvedi (subodh1810) Sep 12, 2023

Choose a reason for hiding this comment

Uh oh!

Prateek Mukhedkar (prateekmukhedkar) Sep 13, 2023

Choose a reason for hiding this comment

Uh oh!

Rodi Reich Zilberman (rodireich) Sep 14, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 12, 2023

source-mysql-strict-encrypt test report (commit b90bd686bd) - ❌

Uh oh!

github-actions Bot commented Sep 12, 2023

source-mysql test report (commit b90bd686bd) - ❌

Uh oh!

github-actions Bot commented Sep 12, 2023

source-mysql-strict-encrypt test report (commit b9c1649a9d) - ❌

Uh oh!

github-actions Bot commented Sep 13, 2023

source-mysql test report (commit b9c1649a9d) - ❌

Uh oh!

github-actions Bot commented Sep 13, 2023

source-mysql test report (commit c93a123e52) - ❌

Uh oh!

github-actions Bot commented Sep 13, 2023

source-mysql-strict-encrypt test report (commit c93a123e52) - ❌

Uh oh!

Prateek Mukhedkar (prateekmukhedkar) commented Sep 13, 2023

Uh oh!

Subodh Chaturvedi (subodh1810) left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 14, 2023

source-mysql test report (commit 0eb24255d0) - ❌

Uh oh!

github-actions Bot commented Sep 14, 2023

source-mysql-strict-encrypt test report (commit 0eb24255d0) - ❌

Uh oh!

Duy Nguyen (nguyenaiden) commented Sep 14, 2023

Uh oh!

Octavia The Approver (octavia-approvington) commented Sep 14, 2023

Uh oh!

github-actions Bot commented Sep 14, 2023

source-mysql test report (commit 8400b75bac) - ❌

Uh oh!

github-actions Bot commented Sep 14, 2023

source-mysql-strict-encrypt test report (commit 8400b75bac) - ❌

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Duy Nguyen (nguyenaiden) commented Sep 12, 2023 •

edited

Loading

github-actions Bot commented Sep 12, 2023 •

edited by nguyenaiden

Loading

source-mysql test report (commit `ff440b155c`) - ❌

source-mysql-strict-encrypt test report (commit `b90bd686bd`) - ❌

source-mysql test report (commit `b90bd686bd`) - ❌

source-mysql-strict-encrypt test report (commit `b9c1649a9d`) - ❌

source-mysql test report (commit `b9c1649a9d`) - ❌

source-mysql test report (commit `c93a123e52`) - ❌

source-mysql-strict-encrypt test report (commit `c93a123e52`) - ❌

source-mysql test report (commit `0eb24255d0`) - ❌

source-mysql-strict-encrypt test report (commit `0eb24255d0`) - ❌

source-mysql test report (commit `8400b75bac`) - ❌

source-mysql-strict-encrypt test report (commit `8400b75bac`) - ❌