Use parse base64Binary to parse binary related data #32610

xiaohansong · 2023-11-16T17:48:15Z

Fix for
https://github.com/airbytehq/oncall/issues/3079
#31311

Rational is

we treat all binary in JDBC as String in discover according to this line:

airbyte/airbyte-cdk/java/airbyte-cdk/core/src/main/java/io/airbyte/cdk/db/jdbc/JdbcSourceOperations.java

Line 147 in da32fc8

case BLOB, BINARY, VARBINARY, LONGVARBINARY -> JsonSchemaType.STRING_BASE_64;

And we are reading it assuming the value is hex: https://github.com/airbytehq/airbyte/blob/da32fc86bf6450b4d02d4cc9e5f761b80529255a[…]airbyte/cdk/db/jdbc/AbstractJdbcCompatibleSourceOperations.java

Tested on mysql connector locally. However this will take effect on all JDBC source connectors too.

Logs output. Please confirm we are going to output base64 encoded string for binary typed columns.

2023-11-16 17:26:26 destination > INFO i.a.i.d.e.l.FirstNLogger(log):27 [2023-11-16T17:25:31.806Z] Users #0001: {"id":"n61enu/ftEkAAAAAAAAAAA==","name":"Alice Smith","email":"alice.smith@example.com","_ab_cdc_updated_at":"2023-11-16T17:25:31.806783963Z","_ab_cdc_log_file":"binlog.000002","_ab_cdc_log_pos":991,"_ab_cdc_deleted_at":null,"_ab_cdc_cursor":170015553100000001}

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user?

For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.

If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:generateScaffolds then checking in your changes
Documentation which references the generator is updated as needed

Updating the Python CDK

Airbyter

Before merging:

Pull Request description explains what problem it is solving
Code change is unit tested
Build and my-py check pass
Smoke test the change on at least one affected connector
- On Github: Run this workflow, passing --use-local-cdk --name=source-<connector> as options
- Locally: airbyte-ci connectors --use-local-cdk --name=source-<connector> test
PR is reviewed and approved

After merging:

Publish the CDK
- The CDK does not follow proper semantic versioning. Choose minor if this the change has significant user impact or is a breaking change. Choose patch otherwise.
- Write a thoughtful changelog message so we know what was updated.
Merge the platform PR that was auto-created for updating the Connector Builder's CDK version
- This step is optional if the change does not affect the connector builder or declarative connectors.

vercel · 2023-11-16T17:48:22Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	❌ Failed (Inspect)			Nov 30, 2023 6:55pm

akashkulk · 2023-11-21T18:17:51Z

airbyte-integrations/connectors/source-mssql-strict-encrypt/build.gradle

@@ -6,7 +6,7 @@ plugins {
 airbyteJavaConnector {
    cdkVersionRequired = '0.4.1'
    features = ['db-sources']
-    useLocalCdk = false
+    useLocalCdk = true


Remember to toggle these to false and bump ckd version finally

akashkulk

Couple of comments:

Let's add a test to MySqlSourceOperationsTest.java
This actually only affects cases where binary data is set as the cursor.

The flow is :

From the source DB, data is read in via putBinary method, which reads it as Base 64 encoded string
Data is saved in the load state, again as a base 64 encoded string
Data is parsed while setting the cursor as a hex string, which this pR fixes. Thanks!

But to clarify setting the cursor field is a source connector only setting - this isn't changing what is sent to the destination in the case of a binary field

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>

xiaohansong · 2023-11-22T21:01:56Z

@akashkulk I'll add the test in the next PR where we point source connector to the new CDK version (otherwise it won't pass because it's still using old logic to parse out binary)

#32746

akashkulk · 2023-11-28T16:44:37Z

...te-cdk/core/src/main/java/io/airbyte/cdk/db/jdbc/AbstractJdbcCompatibleSourceOperations.java

@@ -222,7 +222,7 @@ protected void setString(final PreparedStatement preparedStatement, final int pa
  }

  protected void setBinary(final PreparedStatement preparedStatement, final int parameterIndex, final String value) throws SQLException {
-    preparedStatement.setBytes(parameterIndex, DatatypeConverter.parseHexBinary(value));
+    preparedStatement.setBytes(parameterIndex, DatatypeConverter.parseBase64Binary(value));


Just FYI, this affects other connectors too, since this is shared code. We should understand which connectors this affects - this would be the classes with source operations that inheirit from this and classes that allow BINARY columns as cursors (which I don't think we have more of)

I should clarify, no need to publish them - more of a documentation exercise

They are called from postgres and mysql, and general JdbcSourceOperations which is inherited by Teradata, Oracle, Cockroach, Db2, Snowflake, Mssql, Redshift.

Indeed we do not allow binary column as cursor, both in individual SourceOperations(Mysql, Postgres) and general JDBCSourceOperations

airbyte/airbyte-cdk/java/airbyte-cdk/core/src/main/java/io/airbyte/cdk/db/jdbc/JdbcUtils.java

Line 67 in da32fc8

public static final Set<JDBCType> ALLOWED_CURSOR_TYPES =

cool, good to know - then this shouldn't affect anything for now other than MySQL

akashkulk · 2023-11-28T16:50:05Z

...s/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSourceOperations.java

@@ -63,6 +69,7 @@ public class MySqlSourceOperations extends AbstractJdbcCompatibleSourceOperation
  private static final Logger LOGGER = LoggerFactory.getLogger(MySqlSourceOperations.class);
  private static final Set<MysqlType> ALLOWED_CURSOR_TYPES = Set.of(TINYINT, TINYINT_UNSIGNED, SMALLINT,
      SMALLINT_UNSIGNED, MEDIUMINT, MEDIUMINT_UNSIGNED, INT, INT_UNSIGNED, BIGINT, BIGINT_UNSIGNED,
+      TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB, BINARY, VARBINARY,


As discussed yesterday, we should not allow BINARY as a cursor type, since it's hard for the user to make sure that this value is increasing (hallmark of a good cursor)

We mainly use it for the initial load (where we checkpoint by PK for the user), so I think we can exclude it here

airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml

docs/integrations/sources/mysql.md

akashkulk · 2023-11-29T16:17:11Z

airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml

@@ -11,7 +11,7 @@ data:
  connectorSubtype: database
  connectorType: source
  definitionId: 435bb9a5-7887-4809-aa58-28c27df0d7ad
-  dockerImageTag: 3.1.8
+  dockerImageTag: 3.1.10


Why 3.1.10 instead of 3.1.9?

3.1.9 was used by my previous PR (The one to extend debezium schema loading time)

akashkulk

Looks good in general - some very minor nits. Approving to unblock!! FYI - I bumped the CDK to 0.5.3....so you're going to have to use 0.5.4

xiaohansong · 2023-11-29T21:52:10Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/7038943992
✅ Successfully published Java CDK version=0.5.4!

xiaohansong added 3 commits November 15, 2023 17:20

test

3142b18

remove debug changes

26081ee

more to reset

80842c4

octavia-squidington-iii added the CDK Connector Development Kit label Nov 16, 2023

xiaohansong requested review from akashkulk and prateekmukhedkar November 16, 2023 17:48

xiaohansong and others added 4 commits November 16, 2023 17:57

Automated Commit - Formatting Changes

79b91c3

test fix

e536fa8

Automated Commit - Formatting Changes

04fff5e

enable local cdk to build a test connector

58e1822

xiaohansong requested a review from a team as a code owner November 20, 2023 22:42

octavia-squidington-iii added area/connectors Connector related issues connectors/source/mysql connectors/source/mssql-strict-encrypt labels Nov 20, 2023

Merge remote-tracking branch 'origin/master' into xiaohan/parse

c0696d9

vercel bot deployed to Preview November 20, 2023 23:24 View deployment

fix for strict encrypt connector to pass test

9d2bb0f

octavia-squidington-iii added the connectors/source/mysql-strict-encrypt label Nov 21, 2023

akashkulk reviewed Nov 21, 2023

View reviewed changes

tolik0 and others added 2 commits November 21, 2023 10:54

🐛 Source S3: Fix discovery for zip file (#32677)

be66ee6

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>

revert mssql change

50d6995

octavia-squidington-iii added area/documentation Improvements or additions to documentation connectors/source/s3 labels Nov 21, 2023

xiaohansong added 2 commits November 22, 2023 13:00

clean up PR

9f41792

Merge remote-tracking branch 'origin/master' into xiaohan/parse

dbe1652

octavia-squidington-iii removed area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Nov 22, 2023

fix ci report

d972ee6

akashkulk reviewed Nov 28, 2023

View reviewed changes

xiaohansong added 4 commits November 28, 2023 09:19

save work

514ae9f

revert test change

50fda34

Merge remote-tracking branch 'origin/master' into xiaohan/parse

ebbf687

save work

b66f15f

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Nov 28, 2023

vercel bot deployed to Preview November 28, 2023 18:09 View deployment

xiaohansong requested a review from akashkulk November 28, 2023 21:55

akashkulk reviewed Nov 29, 2023

View reviewed changes

airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml Outdated Show resolved Hide resolved

akashkulk reviewed Nov 29, 2023

View reviewed changes

docs/integrations/sources/mysql.md Outdated Show resolved Hide resolved

akashkulk reviewed Nov 29, 2023

View reviewed changes

akashkulk approved these changes Nov 29, 2023

View reviewed changes

Merge remote-tracking branch 'origin/master' into xiaohan/parse

a19b99c

vercel bot deployed to Preview November 29, 2023 20:02 View deployment

xiaohansong added 2 commits November 29, 2023 14:37

use latest cdk

02cab7f

Merge branch 'master' into xiaohan/parse

66e54db

vercel bot deployed to Preview November 29, 2023 23:41 View deployment

Automated Commit - Formatting Changes

11aecfb

octavia-squidington-iii added the connectors/source/google-search-console label Nov 29, 2023

octavia-squidington-iv requested a review from a team November 30, 2023 00:01

Merge branch 'master' into xiaohan/parse

96ddb8e

vercel bot deployed to Preview November 30, 2023 00:23 View deployment

xiaohansong enabled auto-merge (squash) November 30, 2023 00:25

Merge remote-tracking branch 'origin/master' into xiaohan/parse

48ba86c

vercel bot had a problem deploying to Preview November 30, 2023 18:55 Failure

xiaohansong merged commit 2d26337 into master Nov 30, 2023
22 of 23 checks passed

xiaohansong deleted the xiaohan/parse branch November 30, 2023 19:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use parse base64Binary to parse binary related data #32610

Use parse base64Binary to parse binary related data #32610

xiaohansong commented Nov 16, 2023 •

edited

Loading

vercel bot commented Nov 16, 2023 •

edited

Loading

akashkulk Nov 21, 2023

akashkulk left a comment

xiaohansong commented Nov 22, 2023 •

edited

Loading

akashkulk Nov 28, 2023

akashkulk Nov 28, 2023

xiaohansong Nov 28, 2023 •

edited

Loading

akashkulk Nov 29, 2023 •

edited

Loading

akashkulk Nov 28, 2023 •

edited

Loading

akashkulk Nov 29, 2023

xiaohansong Nov 29, 2023

akashkulk left a comment

xiaohansong commented Nov 29, 2023 •

edited by github-actions bot

Loading

Use parse base64Binary to parse binary related data #32610

Use parse base64Binary to parse binary related data #32610

Conversation

xiaohansong commented Nov 16, 2023 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Airbyter

vercel bot commented Nov 16, 2023 • edited Loading

akashkulk Nov 21, 2023

Choose a reason for hiding this comment

akashkulk left a comment

Choose a reason for hiding this comment

xiaohansong commented Nov 22, 2023 • edited Loading

akashkulk Nov 28, 2023

Choose a reason for hiding this comment

akashkulk Nov 28, 2023

Choose a reason for hiding this comment

xiaohansong Nov 28, 2023 • edited Loading

Choose a reason for hiding this comment

akashkulk Nov 29, 2023 • edited Loading

Choose a reason for hiding this comment

akashkulk Nov 28, 2023 • edited Loading

Choose a reason for hiding this comment

akashkulk Nov 29, 2023

Choose a reason for hiding this comment

xiaohansong Nov 29, 2023

Choose a reason for hiding this comment

akashkulk left a comment

Choose a reason for hiding this comment

xiaohansong commented Nov 29, 2023 • edited by github-actions bot Loading

xiaohansong commented Nov 16, 2023 •

edited

Loading

vercel bot commented Nov 16, 2023 •

edited

Loading

xiaohansong commented Nov 22, 2023 •

edited

Loading

xiaohansong Nov 28, 2023 •

edited

Loading

akashkulk Nov 29, 2023 •

edited

Loading

akashkulk Nov 28, 2023 •

edited

Loading

xiaohansong commented Nov 29, 2023 •

edited by github-actions bot

Loading