🐛Source-mssql: added support for missed data types by etsybaev · Pull Request #9094 · airbytehq/airbyte

Eugene (etsybaev) · 2021-12-23T20:59:06Z

What

We had lot's of types that were not handled properly by the Source-MsSql connector.

How

Updated to connector to properly process data type for a regular connection and for some of CDC connections.

The most of type are handled using different kind of native approaches.
But there is no support for the "hierarchyid" type even in the native SQL Server jdbc driver in DataTypes.
The DB stores it in it's own representation converted to HEX format.

The only way to get it as a text is to query it using internal function like this: "test_column.ToString() as test_column".
So in the final solution we make a pre-request for fetch a first row to get actual types and then wrap required fields with toString() function in eventual request: "test_column" -> "test_column.ToString() as test_column".

Also tested using UI for both full refresh and incremental sync types:
Full refresh:

![Selection_347](https://user-images.githubusercontent.com/4688354/148544339-e12fad8c-a792-403f-a5a7-94caa8a4f246.png

)

Incremental

:

Also checked for cases with empty tables:

So lots of different tests, no fails

🚨 User Impact 🚨

🚨🚨 As we not handle values properly, some of them may come represented in another format, ex. number<->string<->date\datetime\time.
So probably full refresh would be required.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Eugene (etsybaev) · 2021-12-23T20:59:56Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1617056868
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1617056868
No Python unittests run

…a-types

Eugene (etsybaev) · 2021-12-24T18:39:49Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1619950571
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1619950571
No Python unittests run

Eugene (etsybaev) · 2021-12-24T22:39:26Z

/test-performance connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1620362278
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1620362278

Oleksandr Sheheda (alexandr-shegeda) · 2021-12-27T10:10:42Z

kindly reminder that we also should bump the version for the source-mssql-strict-encrypt connector

Eugene (etsybaev) · 2021-12-27T11:27:20Z

kindly reminder that we also should bump the version for the source-mssql-strict-encrypt connector

Sure, will do that along with changelog updating after got all approvals right before merging

…a-types

Eugene (etsybaev) · 2021-12-27T15:41:23Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1627489308
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1627489308
No Python unittests run

Eugene (etsybaev) · 2022-01-06T22:36:00Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1665185321
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1665185321
No Python unittests run

Eugene (etsybaev) · 2022-01-06T22:36:15Z

/test connector=connectors/source-mssql-strict-encrypt

🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1665186326
✅ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1665186326
No Python unittests run

Eugene (etsybaev) · 2022-01-06T22:37:17Z

/test-performance connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1665189440
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1665189440

…a-types # Conflicts: # docs/integrations/sources/mssql.md

Eugene (etsybaev) · 2022-01-07T15:17:06Z

Currently, we have a known issue with build on CI, but locally all checks passed. So going to release new builds

Eugene (etsybaev) · 2022-01-07T15:18:07Z

/publish connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1668110292
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1668110292

Eugene (etsybaev) · 2022-01-07T16:21:34Z

/publish connector=connectors/source-mssql-strict-encrypt

🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1668352736
❌ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1668352736

Eugene (etsybaev) · 2022-01-07T16:28:19Z

/publish connector=connectors/source-mssql-strict-encrypt

🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1668377415
✅ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1668377415

[7728] Source-mssql: added support for missed data types

308c5c6

github-actions Bot added the area/connectors Connector related issues label Dec 23, 2021

Eugene (etsybaev) temporarily deployed to more-secrets December 23, 2021 20:59 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets December 23, 2021 21:01 Inactive

added more fixes + codestyle

5bf749b

Eugene (etsybaev) temporarily deployed to more-secrets December 24, 2021 17:48 Inactive

Merge branch 'master' into etsybaev/7728-source-mssql-support-all-dat…

f403739

…a-types

Eugene (etsybaev) requested a review from Oleksandr Sheheda (alexandr-shegeda) December 24, 2021 18:09

Eugene (etsybaev) temporarily deployed to more-secrets December 24, 2021 18:12 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets December 24, 2021 18:41 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets December 24, 2021 22:41 Inactive

Eugene (etsybaev) linked an issue Dec 26, 2021 that may be closed by this pull request

Support all types in SQL Server source #7728

Closed

Oleksandr Sheheda (alexandr-shegeda) reviewed Dec 27, 2021

View reviewed changes

Comment thread airbyte-db/lib/src/main/java/io/airbyte/db/DataTypeUtils.java Outdated

Oleksandr Sheheda (alexandr-shegeda) reviewed Dec 27, 2021

View reviewed changes

Comment thread .../bases/debezium/src/main/java/io/airbyte/integrations/debezium/internals/MSSQLConverter.java Outdated

Oleksandr Sheheda (alexandr-shegeda) reviewed Dec 27, 2021

View reviewed changes

Comment thread ...snowflake/src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeDatabase.java

Oleksandr Sheheda (alexandr-shegeda) requested a review from Liren Tu (tuliren) December 27, 2021 10:11

Eugene (etsybaev) added 2 commits December 27, 2021 15:01

Merge branch 'master' into etsybaev/7728-source-mssql-support-all-dat…

4e0cd29

…a-types

fixed PR as per comments in PR

f84de55

Eugene (etsybaev) temporarily deployed to more-secrets December 27, 2021 14:06 Inactive

Eugene (etsybaev) changed the title ~~[DRAFT_PR_IGNORE_IT] Source-mssql: added support for missed data types~~ [DRAFT_PR] Source-mssql: added support for missed data types Dec 27, 2021

Eugene (etsybaev) changed the title ~~[DRAFT_PR] Source-mssql: added support for missed data types~~ Source-mssql: added support for missed data types Dec 27, 2021

Jared Rhizor (jrhizor) temporarily deployed to more-secrets December 27, 2021 15:45 Inactive

Eugene (etsybaev) requested review from VitaliiMaltsev and Oleksandr Sheheda (alexandr-shegeda) December 28, 2021 15:44

Jared Rhizor (jrhizor) temporarily deployed to more-secrets January 6, 2022 22:38 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets January 6, 2022 22:39 Inactive

bumped versions and updated changelog

fc0a512

github-actions Bot added the area/documentation Improvements or additions to documentation label Jan 7, 2022

Eugene (etsybaev) changed the title ~~Source-mssql: added support for missed data types~~ 🐛Source-mssql: added support for missed data types Jan 7, 2022

Eugene (etsybaev) temporarily deployed to more-secrets January 7, 2022 12:47 Inactive

Eugene (etsybaev) added 2 commits January 7, 2022 14:52

Merge branch 'master' into etsybaev/7728-source-mssql-support-all-dat…

521129f

…a-types # Conflicts: # docs/integrations/sources/mssql.md

fixed merge-conflict

27fa135

Eugene (etsybaev) temporarily deployed to more-secrets January 7, 2022 14:02 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets January 7, 2022 15:20 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets January 7, 2022 16:23 Inactive

fixed merge-conflict1

2fcac1d

Eugene (etsybaev) temporarily deployed to more-secrets January 7, 2022 16:28 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets January 7, 2022 16:31 Inactive

Updated spec resources

a973210

Eugene (etsybaev) temporarily deployed to more-secrets January 7, 2022 16:45 Inactive

Eugene (etsybaev) merged commit 1877e2a into master Jan 7, 2022

Eugene (etsybaev) deleted the etsybaev/7728-source-mssql-support-all-data-types branch January 7, 2022 17:17

This was referenced Jan 7, 2022

Bump Airbyte version from 0.35.4-alpha to 0.35.5-alpha #9359

Closed

Bump Airbyte version from 0.35.4-alpha to 0.35.5-alpha #9397

Closed

Bump Airbyte version from 0.35.4-alpha to 0.35.5-alpha #9421

Merged

Conversation

Eugene (etsybaev) commented Dec 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Uh oh!

Eugene (etsybaev) commented Dec 23, 2021 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eugene (etsybaev) commented Dec 24, 2021 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eugene (etsybaev) commented Dec 24, 2021 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Oleksandr Sheheda (alexandr-shegeda) commented Dec 27, 2021

Uh oh!

Eugene (etsybaev) commented Dec 27, 2021

Uh oh!

Eugene (etsybaev) commented Dec 27, 2021 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eugene (etsybaev) commented Jan 6, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eugene (etsybaev) commented Jan 6, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eugene (etsybaev) commented Jan 6, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eugene (etsybaev) commented Jan 7, 2022

Uh oh!

Eugene (etsybaev) commented Jan 7, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eugene (etsybaev) commented Jan 7, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eugene (etsybaev) commented Jan 7, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Eugene (etsybaev) commented Dec 23, 2021 •

edited

Loading

Eugene (etsybaev) commented Dec 23, 2021 •

edited by github-actions Bot

Loading

Eugene (etsybaev) commented Dec 24, 2021 •

edited by github-actions Bot

Loading

Eugene (etsybaev) commented Dec 24, 2021 •

edited by github-actions Bot

Loading

Eugene (etsybaev) commented Dec 27, 2021 •

edited by github-actions Bot

Loading

Eugene (etsybaev) commented Jan 6, 2022 •

edited by github-actions Bot

Loading

Eugene (etsybaev) commented Jan 6, 2022 •

edited by github-actions Bot

Loading

Eugene (etsybaev) commented Jan 6, 2022 •

edited by github-actions Bot

Loading

Eugene (etsybaev) commented Jan 7, 2022 •

edited by github-actions Bot

Loading

Eugene (etsybaev) commented Jan 7, 2022 •

edited by github-actions Bot

Loading

Eugene (etsybaev) commented Jan 7, 2022 •

edited by github-actions Bot

Loading