MySql Source: Fix data processing by DoNotPanicUA · Pull Request #6093 · airbytehq/airbyte

Andrii Leonets (DoNotPanicUA) · 2021-09-15T15:13:58Z

What

Fix MySql source fails and wrong data type handlers.

Note that some cases are still relevant as driver restriction or another issue:

Zero-date values are not allowed for the JDBC drivers
CDC process the bool MySQL type as a common boolean logic (>0 -> true). Debezium driver has an issue that prevents the implementation of a handler for the bool type. (Bool field treat as an int in the Debezium handler. As result you are not able to identify that you are processing bool type)
Default MySql (non-CDC) has an issue with processing Binary values. MySql source : source binary node transforms to text node #5878

How

Implement MySql specific handlers for problem data types

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

…nd use common type mapping.

…tils-rework

…fix-mysql # Conflicts: # airbyte-db/lib/src/main/java/io/airbyte/db/SourceOperations.java # airbyte-db/lib/src/main/java/io/airbyte/db/jdbc/JdbcSourceOperations.java

…fix-mysql

Sherif A. Nada (sherifnada) · 2021-09-22T04:45:25Z

Andrii Leonets (@DoNotPanicUA) I won't be able to get to this for 1 or 2 days, my apologies. Will leave a review in a couple of days

…fix-mysql # Conflicts: # docs/integrations/sources/mysql.md

…avior=convertToNull"

…fix-mysql # Conflicts: # docs/integrations/sources/mysql.md

Andrii Leonets (DoNotPanicUA) · 2021-10-11T19:58:26Z

/test connector=source-mysql

🕑 source-mysql https://github.com/airbytehq/airbyte/actions/runs/1330391161
✅ source-mysql https://github.com/airbytehq/airbyte/actions/runs/1330391161
No Python unittests run

Sherif A. Nada (sherifnada) · 2021-10-12T06:45:25Z

        config.get("port").asText(),
        config.get("database").asText()));

+    jdbcUrl.append("?zeroDateTimeBehavior=convertToNull");


we should add a note in the .md docs saying that zero date time is converted to null

Sherif A. Nada (sherifnada) · 2021-10-12T06:49:05Z

            .airbyteType(JsonSchemaPrimitive.NUMBER)
-            .addInsertValues("null")
-            .addNullExpectedValue()
+            .addInsertValues("null", "10.5")


sounds good, thanks!

…a/io/airbyte/integrations/destination/mysql/MySQLDestination.java Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

…fix-mysql

…to aleonets/3931-3932-fix-mysql

Andrii Leonets (DoNotPanicUA) · 2021-10-12T14:40:41Z

/publish connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1333585494
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1333585494

* move logic from static JdbcUtils to JdbcSourceOperations * format * Add methods for all types in order to have possibility rewrite them and use common type mapping. * Make BigQuery in line impl with JDBC changes * fix binary type * add database creations methods with sourceOperations input * add MySqlSourceOperations * fix processing * Fix CDC processing * format * add some tests for double and float * incr version * airbytehq#3931 airbytehq#3932 added zero-date converting to null param "zeroDateTimeBehavior=convertToNull" * remove old tests covered by data type tests + incr ver * Update airbyte-integrations/connectors/destination-mysql/src/main/java/io/airbyte/integrations/destination/mysql/MySQLDestination.java Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * Update docs/integrations/sources/mysql.md Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * add back comments to the data type mapping * incr config version Co-authored-by: Oleksandr Sheheda <alexandrshegeda@gmail.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Andrii Leonets (DoNotPanicUA) added 11 commits September 1, 2021 16:59

move logic from static JdbcUtils to JdbcSourceOperations

88d2191

format

f240e89

Add methods for all types in order to have possibility rewrite them a…

2695b29

…nd use common type mapping.

Make BigQuery in line impl with JDBC changes

a6722d6

fix binary type

19edc8f

Merge remote-tracking branch 'origin/master' into aleonets/5704-jdbcu…

df74b34

…tils-rework

add database creations methods with sourceOperations input

c4c204c

add MySqlSourceOperations

a0670ac

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

e9c96eb

…fix-mysql # Conflicts: # airbyte-db/lib/src/main/java/io/airbyte/db/SourceOperations.java # airbyte-db/lib/src/main/java/io/airbyte/db/jdbc/JdbcSourceOperations.java

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

534dbac

…fix-mysql

fix processing

7703193

Andrii Leonets (DoNotPanicUA) self-assigned this Sep 15, 2021

github-actions Bot added the area/connectors Connector related issues label Sep 15, 2021

Andrii Leonets (DoNotPanicUA) changed the title ~~MySql Source: Fix data processing~~ 🐛 MySql Source: Fix data processing Sep 17, 2021

Andrii Leonets (DoNotPanicUA) changed the title ~~🐛 MySql Source: Fix data processing~~ bug MySql Source: Fix data processing Sep 17, 2021

Andrii Leonets (DoNotPanicUA) changed the title ~~bug MySql Source: Fix data processing~~ MySql Source: Fix data processing Sep 17, 2021

Fix CDC processing

dece86e

github-actions Bot added the area/documentation Improvements or additions to documentation label Sep 17, 2021

format

86df40c

Andrii Leonets (DoNotPanicUA) requested a review from Oleksandr Sheheda (alexandr-shegeda) September 17, 2021 21:23

add some tests for double and float

35c5781

Oleksandr Sheheda (alexandr-shegeda) approved these changes Sep 21, 2021

View reviewed changes

Andrii Leonets (DoNotPanicUA) requested review from Sherif A. Nada (sherifnada) and Subodh Chaturvedi (subodh1810) September 21, 2021 13:01

Andrii Leonets (DoNotPanicUA) marked this pull request as ready for review September 21, 2021 13:02

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets September 23, 2021 12:25 Inactive

Andrii Leonets (DoNotPanicUA) added 2 commits September 23, 2021 16:30

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

ea4e2cf

…fix-mysql # Conflicts: # docs/integrations/sources/mysql.md

incr version

c4c4e85

Andrii Leonets (DoNotPanicUA) force-pushed the aleonets/3931-3932-fix-mysql branch from 3a6bc38 to c4c4e85 Compare September 23, 2021 13:31

Oleksandr Sheheda (alexandr-shegeda) linked an issue Oct 4, 2021 that may be closed by this pull request

Source MySQL: Connector fails if source contains "tricky" date values #3931

Closed

Oleksandr Sheheda (alexandr-shegeda) added 2 commits October 4, 2021 15:59

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

f1d3b7c

…fix-mysql # Conflicts: # docs/integrations/sources/mysql.md

#3931 #3932 added zero-date converting to null param "zeroDateTimeBeh…

0f0883b

…avior=convertToNull"

Oleksandr Sheheda (alexandr-shegeda) temporarily deployed to more-secrets October 6, 2021 12:37 Inactive

Oleksandr Sheheda (alexandr-shegeda) temporarily deployed to more-secrets October 11, 2021 15:04 Inactive

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

1d6b02a

…fix-mysql # Conflicts: # docs/integrations/sources/mysql.md

Andrii Leonets (DoNotPanicUA) mentioned this pull request Oct 11, 2021

[EPIC] Add possibility to run datatype tests using different configurations #6961

Closed

remove old tests covered by data type tests + incr ver

ad88abc

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets October 11, 2021 19:19 Inactive

Andrii Leonets (DoNotPanicUA) requested a review from Sherif A. Nada (sherifnada) October 11, 2021 19:55

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets October 11, 2021 19:57 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets October 11, 2021 20:01 Inactive

Sherif A. Nada (sherifnada) approved these changes Oct 12, 2021

View reviewed changes

Andrii Leonets (DoNotPanicUA) and others added 2 commits October 12, 2021 14:38

Update airbyte-integrations/connectors/destination-mysql/src/main/jav…

11005ce

…a/io/airbyte/integrations/destination/mysql/MySQLDestination.java Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Update docs/integrations/sources/mysql.md

8d51497

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets October 12, 2021 11:41 Inactive

Andrii Leonets (DoNotPanicUA) added 3 commits October 12, 2021 15:26

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

b7a4d28

…fix-mysql

add back comments to the data type mapping

342b8f1

Merge remote-tracking branch 'origin/aleonets/3931-3932-fix-mysql' in…

b73787c

…to aleonets/3931-3932-fix-mysql

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets October 12, 2021 12:56 Inactive

incr config version

a166c6a

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets October 12, 2021 14:40 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets October 12, 2021 14:43 Inactive

Andrii Leonets (DoNotPanicUA) merged commit a98b3fe into master Oct 12, 2021

Andrii Leonets (DoNotPanicUA) deleted the aleonets/3931-3932-fix-mysql branch October 12, 2021 15:08

Jared Rhizor (jrhizor) mentioned this pull request Oct 13, 2021

Bump Airbyte version from 0.30.18-alpha to 0.30.19-alpha #6991

Merged

Rodi Reich Zilberman (rodireich) mentioned this pull request Apr 12, 2023

Source MySQL: java.sql.SQLException: MONTH #18609

Closed

Navigation Menu

Uh oh!

MySql Source: Fix data processing - #6093

MySql Source: Fix data processing#6093
Andrii Leonets (DoNotPanicUA) merged 26 commits into
masterfrom
aleonets/3931-3932-fix-mysql

Andrii Leonets (DoNotPanicUA) commented Sep 15, 2021 •

edited

Loading

Uh oh!

Sherif A. Nada (sherifnada) commented Sep 22, 2021

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Oct 11, 2021 •

edited by github-actions Bot

Loading

Uh oh!

Uh oh!

Sherif A. Nada (sherifnada) Oct 12, 2021

Uh oh!

Uh oh!

Sherif A. Nada (sherifnada) Oct 12, 2021

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Oct 12, 2021 •

edited by github-actions Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

Andrii Leonets (DoNotPanicUA) commented Sep 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Uh oh!

Sherif A. Nada (sherifnada) commented Sep 22, 2021

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Oct 11, 2021 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Sherif A. Nada (sherifnada) Oct 12, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Sherif A. Nada (sherifnada) Oct 12, 2021

Choose a reason for hiding this comment

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Oct 12, 2021 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Andrii Leonets (DoNotPanicUA) commented Sep 15, 2021 •

edited

Loading

Andrii Leonets (DoNotPanicUA) commented Oct 11, 2021 •

edited by github-actions Bot

Loading

Andrii Leonets (DoNotPanicUA) commented Oct 12, 2021 •

edited by github-actions Bot

Loading