MySql Source: Fix data processing #6093

DoNotPanicUA · 2021-09-15T15:13:58Z

What

Fix MySql source fails and wrong data type handlers.

Note that some cases are still relevant as driver restriction or another issue:

Zero-date values are not allowed for the JDBC drivers
CDC process the bool MySQL type as a common boolean logic (>0 -> true). Debezium driver has an issue that prevents the implementation of a handler for the bool type. (Bool field treat as an int in the Debezium handler. As result you are not able to identify that you are processing bool type)
Default MySql (non-CDC) has an issue with processing Binary values. MySql source : source binary node transforms to text node #5878

How

Implement MySql specific handlers for problem data types

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

…nd use common type mapping.

…tils-rework

…fix-mysql # Conflicts: # airbyte-db/lib/src/main/java/io/airbyte/db/SourceOperations.java # airbyte-db/lib/src/main/java/io/airbyte/db/jdbc/JdbcSourceOperations.java

…fix-mysql

sherifnada · 2021-09-22T04:45:25Z

@DoNotPanicUA I won't be able to get to this for 1 or 2 days, my apologies. Will leave a review in a couple of days

…fix-mysql # Conflicts: # docs/integrations/sources/mysql.md

…avior=convertToNull"

…fix-mysql # Conflicts: # docs/integrations/sources/mysql.md

DoNotPanicUA · 2021-10-11T19:58:26Z

/test connector=source-mysql

🕑 source-mysql https://github.com/airbytehq/airbyte/actions/runs/1330391161
✅ source-mysql https://github.com/airbytehq/airbyte/actions/runs/1330391161
No Python unittests run

docs/integrations/sources/mysql.md

sherifnada · 2021-10-12T06:45:25Z

...tination-mysql/src/main/java/io/airbyte/integrations/destination/mysql/MySQLDestination.java

@@ -87,8 +87,9 @@ public JsonNode toJdbcConfig(JsonNode config) {
        config.get("port").asText(),
        config.get("database").asText()));

+    jdbcUrl.append("?zeroDateTimeBehavior=convertToNull");


we should add a note in the .md docs saying that zero date time is converted to null

...tination-mysql/src/main/java/io/airbyte/integrations/destination/mysql/MySQLDestination.java

sherifnada · 2021-10-12T06:49:05Z

.../src/test-integration/java/io/airbyte/integrations/source/mysql/MySqlSourceDatatypeTest.java

@@ -175,16 +176,16 @@ protected void initTests() {
        TestDataHolder.builder()
            .sourceType("float")
            .airbyteType(JsonSchemaPrimitive.NUMBER)
-            .addInsertValues("null")
-            .addNullExpectedValue()
+            .addInsertValues("null", "10.5")


sounds good, thanks!

…a/io/airbyte/integrations/destination/mysql/MySQLDestination.java Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

…fix-mysql

…to aleonets/3931-3932-fix-mysql

DoNotPanicUA · 2021-10-12T14:40:41Z

/publish connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1333585494
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1333585494

* move logic from static JdbcUtils to JdbcSourceOperations * format * Add methods for all types in order to have possibility rewrite them and use common type mapping. * Make BigQuery in line impl with JDBC changes * fix binary type * add database creations methods with sourceOperations input * add MySqlSourceOperations * fix processing * Fix CDC processing * format * add some tests for double and float * incr version * airbytehq#3931 airbytehq#3932 added zero-date converting to null param "zeroDateTimeBehavior=convertToNull" * remove old tests covered by data type tests + incr ver * Update airbyte-integrations/connectors/destination-mysql/src/main/java/io/airbyte/integrations/destination/mysql/MySQLDestination.java Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * Update docs/integrations/sources/mysql.md Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * add back comments to the data type mapping * incr config version Co-authored-by: Oleksandr Sheheda <alexandrshegeda@gmail.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

DoNotPanicUA added 11 commits September 1, 2021 16:59

move logic from static JdbcUtils to JdbcSourceOperations

88d2191

format

f240e89

Add methods for all types in order to have possibility rewrite them a…

2695b29

…nd use common type mapping.

Make BigQuery in line impl with JDBC changes

a6722d6

fix binary type

19edc8f

Merge remote-tracking branch 'origin/master' into aleonets/5704-jdbcu…

df74b34

…tils-rework

add database creations methods with sourceOperations input

c4c204c

add MySqlSourceOperations

a0670ac

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

e9c96eb

…fix-mysql # Conflicts: # airbyte-db/lib/src/main/java/io/airbyte/db/SourceOperations.java # airbyte-db/lib/src/main/java/io/airbyte/db/jdbc/JdbcSourceOperations.java

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

534dbac

…fix-mysql

fix processing

7703193

DoNotPanicUA self-assigned this Sep 15, 2021

github-actions bot added the area/connectors Connector related issues label Sep 15, 2021

DoNotPanicUA changed the title ~~MySql Source: Fix data processing~~ 🐛 MySql Source: Fix data processing Sep 17, 2021

DoNotPanicUA changed the title ~~🐛 MySql Source: Fix data processing~~ bug MySql Source: Fix data processing Sep 17, 2021

DoNotPanicUA changed the title ~~bug MySql Source: Fix data processing~~ MySql Source: Fix data processing Sep 17, 2021

Fix CDC processing

dece86e

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 17, 2021

format

86df40c

DoNotPanicUA requested a review from alexandr-shegeda September 17, 2021 21:23

add some tests for double and float

35c5781

alexandr-shegeda approved these changes Sep 21, 2021

View reviewed changes

DoNotPanicUA requested review from subodh1810 and sherifnada September 21, 2021 13:01

DoNotPanicUA marked this pull request as ready for review September 21, 2021 13:02

DoNotPanicUA temporarily deployed to more-secrets September 23, 2021 12:25 Inactive

DoNotPanicUA added 2 commits September 23, 2021 16:30

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

ea4e2cf

…fix-mysql # Conflicts: # docs/integrations/sources/mysql.md

incr version

c4c4e85

DoNotPanicUA force-pushed the aleonets/3931-3932-fix-mysql branch from 3a6bc38 to c4c4e85 Compare September 23, 2021 13:31

alexandr-shegeda linked an issue Oct 4, 2021 that may be closed by this pull request

Source MySQL: Connector fails if source contains "tricky" date values #3931

Closed

alexandr-shegeda added 2 commits October 4, 2021 15:59

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

f1d3b7c

…fix-mysql # Conflicts: # docs/integrations/sources/mysql.md

#3931 #3932 added zero-date converting to null param "zeroDateTimeBeh…

0f0883b

…avior=convertToNull"

alexandr-shegeda temporarily deployed to more-secrets October 6, 2021 12:37 Inactive

alexandr-shegeda temporarily deployed to more-secrets October 11, 2021 15:04 Inactive

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

1d6b02a

…fix-mysql # Conflicts: # docs/integrations/sources/mysql.md

DoNotPanicUA mentioned this pull request Oct 11, 2021

[EPIC] Add possibility to run datatype tests using different configurations #6961

Closed

remove old tests covered by data type tests + incr ver

ad88abc

DoNotPanicUA temporarily deployed to more-secrets October 11, 2021 19:19 Inactive

DoNotPanicUA requested a review from sherifnada October 11, 2021 19:55

DoNotPanicUA temporarily deployed to more-secrets October 11, 2021 19:57 Inactive

jrhizor temporarily deployed to more-secrets October 11, 2021 20:01 Inactive

sherifnada approved these changes Oct 12, 2021

View reviewed changes

DoNotPanicUA and others added 2 commits October 12, 2021 14:38

Update airbyte-integrations/connectors/destination-mysql/src/main/jav…

11005ce

…a/io/airbyte/integrations/destination/mysql/MySQLDestination.java Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Update docs/integrations/sources/mysql.md

8d51497

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

DoNotPanicUA temporarily deployed to more-secrets October 12, 2021 11:41 Inactive

DoNotPanicUA added 3 commits October 12, 2021 15:26

Merge remote-tracking branch 'origin/master' into aleonets/3931-3932-…

b7a4d28

…fix-mysql

add back comments to the data type mapping

342b8f1

Merge remote-tracking branch 'origin/aleonets/3931-3932-fix-mysql' in…

b73787c

…to aleonets/3931-3932-fix-mysql

DoNotPanicUA temporarily deployed to more-secrets October 12, 2021 12:56 Inactive

incr config version

a166c6a

DoNotPanicUA temporarily deployed to more-secrets October 12, 2021 14:40 Inactive

jrhizor temporarily deployed to more-secrets October 12, 2021 14:43 Inactive

DoNotPanicUA merged commit a98b3fe into master Oct 12, 2021

DoNotPanicUA deleted the aleonets/3931-3932-fix-mysql branch October 12, 2021 15:08

jrhizor mentioned this pull request Oct 13, 2021

Bump Airbyte version from 0.30.18-alpha to 0.30.19-alpha #6991

Merged

rodireich mentioned this pull request Apr 12, 2023

Source MySQL: java.sql.SQLException: MONTH #18609

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MySql Source: Fix data processing #6093

MySql Source: Fix data processing #6093

DoNotPanicUA commented Sep 15, 2021 •

edited

Loading

sherifnada commented Sep 22, 2021

DoNotPanicUA commented Oct 11, 2021 •

edited by github-actions bot

Loading

sherifnada Oct 12, 2021

sherifnada Oct 12, 2021

DoNotPanicUA commented Oct 12, 2021 •

edited by github-actions bot

Loading

MySql Source: Fix data processing #6093

MySql Source: Fix data processing #6093

Conversation

DoNotPanicUA commented Sep 15, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

sherifnada commented Sep 22, 2021

DoNotPanicUA commented Oct 11, 2021 • edited by github-actions bot Loading

sherifnada Oct 12, 2021

Choose a reason for hiding this comment

sherifnada Oct 12, 2021

Choose a reason for hiding this comment

DoNotPanicUA commented Oct 12, 2021 • edited by github-actions bot Loading

DoNotPanicUA commented Sep 15, 2021 •

edited

Loading

DoNotPanicUA commented Oct 11, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Oct 12, 2021 •

edited by github-actions bot

Loading