🐛Source-mssql: added support for missed data types #9094

etsybaev · 2021-12-23T20:59:06Z

What

We had lot's of types that were not handled properly by the Source-MsSql connector.

How

Updated to connector to properly process data type for a regular connection and for some of CDC connections.

The most of type are handled using different kind of native approaches.
But there is no support for the "hierarchyid" type even in the native SQL Server jdbc driver in DataTypes.
The DB stores it in it's own representation converted to HEX format.

The only way to get it as a text is to query it using internal function like this: "test_column.ToString() as test_column".
So in the final solution we make a pre-request for fetch a first row to get actual types and then wrap required fields with toString() function in eventual request: "test_column" -> "test_column.ToString() as test_column".

Also tested using UI for both full refresh and incremental sync types:
Full refresh:

![Selection_347](https://user-images.githubusercontent.com/4688354/148544339-e12fad8c-a792-403f-a5a7-94caa8a4f246.png

)

Incremental

:

Also checked for cases with empty tables:

So lots of different tests, no fails

🚨 User Impact 🚨

🚨🚨 As we not handle values properly, some of them may come represented in another format, ex. number<->string<->date\datetime\time.
So probably full refresh would be required.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

etsybaev · 2021-12-23T20:59:56Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1617056868
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1617056868
No Python unittests run

…a-types

etsybaev · 2021-12-24T18:39:49Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1619950571
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1619950571
No Python unittests run

etsybaev · 2021-12-24T22:39:26Z

/test-performance connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1620362278
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1620362278

airbyte-db/lib/src/main/java/io/airbyte/db/DataTypeUtils.java

.../bases/debezium/src/main/java/io/airbyte/integrations/debezium/internals/MSSQLConverter.java

...snowflake/src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeDatabase.java

alexandr-shegeda · 2021-12-27T10:10:42Z

kindly reminder that we also should bump the version for the source-mssql-strict-encrypt connector

etsybaev · 2021-12-27T11:27:20Z

kindly reminder that we also should bump the version for the source-mssql-strict-encrypt connector

Sure, will do that along with changelog updating after got all approvals right before merging

…a-types

etsybaev · 2021-12-27T15:41:23Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1627489308
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1627489308
No Python unittests run

etsybaev · 2022-01-06T22:36:00Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1665185321
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1665185321
No Python unittests run

etsybaev · 2022-01-06T22:36:15Z

/test connector=connectors/source-mssql-strict-encrypt

🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1665186326
✅ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1665186326
No Python unittests run

etsybaev · 2022-01-06T22:37:17Z

/test-performance connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1665189440
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1665189440

…a-types # Conflicts: # docs/integrations/sources/mssql.md

etsybaev · 2022-01-07T15:17:06Z

Currently, we have a known issue with build on CI, but locally all checks passed. So going to release new builds

etsybaev · 2022-01-07T15:18:07Z

/publish connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1668110292
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1668110292

etsybaev · 2022-01-07T16:21:34Z

/publish connector=connectors/source-mssql-strict-encrypt

🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1668352736
❌ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1668352736

etsybaev · 2022-01-07T16:28:19Z

/publish connector=connectors/source-mssql-strict-encrypt

🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1668377415
✅ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1668377415

[7728] Source-mssql: added support for missed data types

308c5c6

github-actions bot added the area/connectors Connector related issues label Dec 23, 2021

etsybaev temporarily deployed to more-secrets December 23, 2021 20:59 Inactive

jrhizor temporarily deployed to more-secrets December 23, 2021 21:01 Inactive

added more fixes + codestyle

5bf749b

etsybaev temporarily deployed to more-secrets December 24, 2021 17:48 Inactive

Merge branch 'master' into etsybaev/7728-source-mssql-support-all-dat…

f403739

…a-types

etsybaev requested a review from alexandr-shegeda December 24, 2021 18:09

etsybaev temporarily deployed to more-secrets December 24, 2021 18:12 Inactive

jrhizor temporarily deployed to more-secrets December 24, 2021 18:41 Inactive

jrhizor temporarily deployed to more-secrets December 24, 2021 22:41 Inactive

etsybaev linked an issue Dec 26, 2021 that may be closed by this pull request

Support all types in SQL Server source #7728

Closed

alexandr-shegeda reviewed Dec 27, 2021

View reviewed changes

airbyte-db/lib/src/main/java/io/airbyte/db/DataTypeUtils.java Outdated Show resolved Hide resolved

alexandr-shegeda reviewed Dec 27, 2021

View reviewed changes

.../bases/debezium/src/main/java/io/airbyte/integrations/debezium/internals/MSSQLConverter.java Outdated Show resolved Hide resolved

alexandr-shegeda reviewed Dec 27, 2021

View reviewed changes

...snowflake/src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeDatabase.java Outdated Show resolved Hide resolved

alexandr-shegeda requested a review from tuliren December 27, 2021 10:11

etsybaev added 2 commits December 27, 2021 15:01

Merge branch 'master' into etsybaev/7728-source-mssql-support-all-dat…

4e0cd29

…a-types

fixed PR as per comments in PR

f84de55

etsybaev temporarily deployed to more-secrets December 27, 2021 14:06 Inactive

etsybaev changed the title ~~[DRAFT_PR_IGNORE_IT] Source-mssql: added support for missed data types~~ [DRAFT_PR] Source-mssql: added support for missed data types Dec 27, 2021

etsybaev changed the title ~~[DRAFT_PR] Source-mssql: added support for missed data types~~ Source-mssql: added support for missed data types Dec 27, 2021

jrhizor temporarily deployed to more-secrets December 27, 2021 15:45 Inactive

etsybaev requested review from VitaliiMaltsev and alexandr-shegeda December 28, 2021 15:44

jrhizor temporarily deployed to more-secrets January 6, 2022 22:38 Inactive

jrhizor temporarily deployed to more-secrets January 6, 2022 22:39 Inactive

bumped versions and updated changelog

fc0a512

github-actions bot added the area/documentation Improvements or additions to documentation label Jan 7, 2022

etsybaev changed the title ~~Source-mssql: added support for missed data types~~ 🐛Source-mssql: added support for missed data types Jan 7, 2022

etsybaev temporarily deployed to more-secrets January 7, 2022 12:47 Inactive

etsybaev added 2 commits January 7, 2022 14:52

Merge branch 'master' into etsybaev/7728-source-mssql-support-all-dat…

521129f

…a-types # Conflicts: # docs/integrations/sources/mssql.md

fixed merge-conflict

27fa135

etsybaev temporarily deployed to more-secrets January 7, 2022 14:02 Inactive

jrhizor temporarily deployed to more-secrets January 7, 2022 15:20 Inactive

jrhizor temporarily deployed to more-secrets January 7, 2022 16:23 Inactive

fixed merge-conflict1

2fcac1d

etsybaev temporarily deployed to more-secrets January 7, 2022 16:28 Inactive

jrhizor temporarily deployed to more-secrets January 7, 2022 16:31 Inactive

Updated spec resources

a973210

etsybaev temporarily deployed to more-secrets January 7, 2022 16:45 Inactive

etsybaev merged commit 1877e2a into master Jan 7, 2022

etsybaev deleted the etsybaev/7728-source-mssql-support-all-data-types branch January 7, 2022 17:17

This was referenced Jan 7, 2022

Bump Airbyte version from 0.35.4-alpha to 0.35.5-alpha #9359

Closed

Bump Airbyte version from 0.35.4-alpha to 0.35.5-alpha #9397

Closed

Bump Airbyte version from 0.35.4-alpha to 0.35.5-alpha #9421

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛Source-mssql: added support for missed data types #9094

🐛Source-mssql: added support for missed data types #9094

etsybaev commented Dec 23, 2021 •

edited

Loading

etsybaev commented Dec 23, 2021 •

edited by github-actions bot

Loading

etsybaev commented Dec 24, 2021 •

edited by github-actions bot

Loading

etsybaev commented Dec 24, 2021 •

edited by github-actions bot

Loading

alexandr-shegeda commented Dec 27, 2021

etsybaev commented Dec 27, 2021

etsybaev commented Dec 27, 2021 •

edited by github-actions bot

Loading

etsybaev commented Jan 6, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 6, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 6, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 7, 2022

etsybaev commented Jan 7, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 7, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 7, 2022 •

edited by github-actions bot

Loading

🐛Source-mssql: added support for missed data types #9094

🐛Source-mssql: added support for missed data types #9094

Conversation

etsybaev commented Dec 23, 2021 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

etsybaev commented Dec 23, 2021 • edited by github-actions bot Loading

etsybaev commented Dec 24, 2021 • edited by github-actions bot Loading

etsybaev commented Dec 24, 2021 • edited by github-actions bot Loading

alexandr-shegeda commented Dec 27, 2021

etsybaev commented Dec 27, 2021

etsybaev commented Dec 27, 2021 • edited by github-actions bot Loading

etsybaev commented Jan 6, 2022 • edited by github-actions bot Loading

etsybaev commented Jan 6, 2022 • edited by github-actions bot Loading

etsybaev commented Jan 6, 2022 • edited by github-actions bot Loading

etsybaev commented Jan 7, 2022

etsybaev commented Jan 7, 2022 • edited by github-actions bot Loading

etsybaev commented Jan 7, 2022 • edited by github-actions bot Loading

etsybaev commented Jan 7, 2022 • edited by github-actions bot Loading

etsybaev commented Dec 23, 2021 •

edited

Loading

etsybaev commented Dec 23, 2021 •

edited by github-actions bot

Loading

etsybaev commented Dec 24, 2021 •

edited by github-actions bot

Loading

etsybaev commented Dec 24, 2021 •

edited by github-actions bot

Loading

etsybaev commented Dec 27, 2021 •

edited by github-actions bot

Loading

etsybaev commented Jan 6, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 6, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 6, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 7, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 7, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 7, 2022 •

edited by github-actions bot

Loading