🐛 Source MSSQL: fix data type (smalldatetime, smallmoney) conversion from mssql source (#5609) #7386

yurii-bidiuk · 2021-10-26T12:50:11Z

What

Fixed data type (smalldatetime, smallmoney) conversion from mssql source (#5609)

How

Created custom MSSQLConverter that implements CustomConverter

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

CLAassistant · 2021-10-26T12:50:16Z

All committers have signed the CLA.

yurii-bidiuk · 2021-10-26T13:01:10Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1385732281
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1385732281
No Python unittests run

...c/main/resources/config/STANDARD_SOURCE_DEFINITION/b5ea17b1-f170-46dc-bc31-cc744ca984c1.json

irynakruk

LGTM

DoNotPanicUA

Good PR.
I'm only worrying about DebeziumConverterUtils.java and possible conflicts.

DoNotPanicUA · 2021-10-26T15:05:11Z

...ebezium/src/main/java/io/airbyte/integrations/debezium/internals/DebeziumConverterUtils.java

+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public final class DebeziumConverterUtils {


There will be a conflict with PR #7339 where DebeziumConverterUtils.java is also added!
This change should be consolidated in order to avoid merge conflicts and any effects on earlier merge code.

Agree. I will apply changes from #7339 once it is merged to master

DoNotPanicUA · 2021-10-26T15:11:31Z

...tors/source-mssql/src/main/java/io/airbyte/integrations/source/mssql/MssqlCdcProperties.java

@@ -26,6 +26,10 @@ static Properties getDebeziumProperties() {
    // https://debezium.io/documentation/reference/1.4/connectors/sqlserver.html#sqlserver-property-provide-transaction-metadata
    props.setProperty("provide.transaction.metadata", "false");

+    props.setProperty("converters", "smalldatetime, smallmoney");
+    props.setProperty("smalldatetime.type", "io.airbyte.integrations.debezium.internals.MSSQLConverter");


you should not put multiply properties for the same converter.
there is no relation between converter name and actual types which are covered by it. Please leave only one.
https://debezium.io/documentation/reference/development/converters.html
P.S. I agree that our names confuse mostly and would be better to use converter name.type instead.

yurii-bidiuk · 2021-10-28T09:30:03Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1393967799
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1393967799
No Python unittests run

#5609)

alexandr-shegeda

LGTM. we can proceed with publish/merge without airbyte review

yurii-bidiuk · 2021-11-04T13:21:01Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1421420518
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1421420518
No Python unittests run

yurii-bidiuk · 2021-11-04T14:46:56Z

/test connector=connectors/source-mssql-strict-encrypt

🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1421778804
✅ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1421778804
No Python unittests run

yurii-bidiuk · 2021-11-04T15:19:28Z

/publish connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1421926615
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1421926615

yurii-bidiuk · 2021-11-04T16:23:15Z

/publish connector=connectors/source-mssql-strict-encrypt

🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1422205024
❌ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1422205024

yurii-bidiuk · 2021-11-04T16:46:38Z

/publish connector=connectors/source-mssql-strict-encrypt

🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1422294704
❌ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1422294704
🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1422294704
❌ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1422294704

…from mssql source (airbytehq#5609) (airbytehq#7386) * Fix data type (smalldatetime, smallmoney) conversion from mssql source (airbytehq#5609) * Fixed code format * Bumb new version * Update documentation (mssql.md) * formating * fixed converter properties * aligned converter utils with airbytehq#7339 Co-authored-by: Andrii Leonets <30464745+DoNotPanicUA@users.noreply.github.com>

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 26, 2021

yurii-bidiuk temporarily deployed to more-secrets October 26, 2021 12:58 Inactive

jrhizor temporarily deployed to more-secrets October 26, 2021 13:03 Inactive

yurii-bidiuk requested a review from alexandr-shegeda October 26, 2021 13:08

yurii-bidiuk mentioned this pull request Oct 26, 2021

Source MSSQL to S3 Failed to convert JSON to Avro #5609

Closed

alexandr-shegeda requested a review from DoNotPanicUA October 26, 2021 13:42

irynakruk linked an issue Oct 26, 2021 that may be closed by this pull request

Source MSSQL to S3 Failed to convert JSON to Avro #5609

Closed

irynakruk self-requested a review October 26, 2021 14:31

irynakruk reviewed Oct 26, 2021

View reviewed changes

...c/main/resources/config/STANDARD_SOURCE_DEFINITION/b5ea17b1-f170-46dc-bc31-cc744ca984c1.json Show resolved Hide resolved

irynakruk approved these changes Oct 26, 2021

View reviewed changes

DoNotPanicUA suggested changes Oct 26, 2021

View reviewed changes

DoNotPanicUA temporarily deployed to more-secrets October 26, 2021 15:15 Inactive

DoNotPanicUA mentioned this pull request Oct 26, 2021

🐛 Fix data type tests in CdcPostgresSourceDatatypeTest #7339

Merged

38 tasks

yurii-bidiuk temporarily deployed to more-secrets October 26, 2021 19:16 Inactive

yurii-bidiuk temporarily deployed to more-secrets October 27, 2021 11:02 Inactive

jrhizor temporarily deployed to more-secrets October 28, 2021 09:32 Inactive

yurii-bidiuk force-pushed the yurii-bidiuk/5609-source-mssql-fix-convert branch from 623f680 to 72172cb Compare November 3, 2021 22:01

yurii-bidiuk temporarily deployed to more-secrets November 3, 2021 22:03 Inactive

yurii-bidiuk requested a review from DoNotPanicUA November 4, 2021 09:33

yurii-bidiuk and others added 7 commits November 4, 2021 11:44

Fix data type (smalldatetime, smallmoney) conversion from mssql source (

1a7f9ea

#5609)

Fixed code format

188b281

Bumb new version

ebb9030

Update documentation (mssql.md)

67b25b5

formating

7157d38

fixed converter properties

b6d68af

aligned converter utils with #7339

8df87e2

yurii-bidiuk force-pushed the yurii-bidiuk/5609-source-mssql-fix-convert branch from 72172cb to 8df87e2 Compare November 4, 2021 09:55

yurii-bidiuk temporarily deployed to more-secrets November 4, 2021 09:57 Inactive

alexandr-shegeda approved these changes Nov 4, 2021

View reviewed changes

yurii-bidiuk temporarily deployed to more-secrets November 4, 2021 10:34 Inactive

yurii-bidiuk temporarily deployed to more-secrets November 4, 2021 12:49 Inactive

DoNotPanicUA approved these changes Nov 4, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets November 4, 2021 13:23 Inactive

jrhizor temporarily deployed to more-secrets November 4, 2021 14:49 Inactive

jrhizor temporarily deployed to more-secrets November 4, 2021 15:21 Inactive

jrhizor temporarily deployed to more-secrets November 4, 2021 16:25 Inactive

jrhizor temporarily deployed to more-secrets November 4, 2021 16:48 Inactive

jrhizor temporarily deployed to more-secrets November 4, 2021 18:13 Inactive

yurii-bidiuk marked this pull request as ready for review November 5, 2021 10:20

yurii-bidiuk merged commit 56db806 into master Nov 5, 2021

yurii-bidiuk deleted the yurii-bidiuk/5609-source-mssql-fix-convert branch November 5, 2021 10:21

jrhizor mentioned this pull request Nov 6, 2021

Bump Airbyte version from 0.30.31-alpha to 0.30.32-alpha #7700

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source MSSQL: fix data type (smalldatetime, smallmoney) conversion from mssql source (#5609) #7386

🐛 Source MSSQL: fix data type (smalldatetime, smallmoney) conversion from mssql source (#5609) #7386

yurii-bidiuk commented Oct 26, 2021 •

edited

Loading

CLAassistant commented Oct 26, 2021 •

edited

Loading

yurii-bidiuk commented Oct 26, 2021 •

edited by github-actions bot

Loading

irynakruk left a comment

DoNotPanicUA left a comment

DoNotPanicUA Oct 26, 2021

yurii-bidiuk Oct 26, 2021

DoNotPanicUA Oct 26, 2021

yurii-bidiuk Oct 26, 2021

yurii-bidiuk commented Oct 28, 2021 •

edited by github-actions bot

Loading

alexandr-shegeda left a comment

yurii-bidiuk commented Nov 4, 2021 •

edited by github-actions bot

Loading

yurii-bidiuk commented Nov 4, 2021 •

edited by github-actions bot

Loading

yurii-bidiuk commented Nov 4, 2021 •

edited by github-actions bot

Loading

yurii-bidiuk commented Nov 4, 2021 •

edited by github-actions bot

Loading

yurii-bidiuk commented Nov 4, 2021 •

edited by github-actions bot

Loading

🐛 Source MSSQL: fix data type (smalldatetime, smallmoney) conversion from mssql source (#5609) #7386

🐛 Source MSSQL: fix data type (smalldatetime, smallmoney) conversion from mssql source (#5609) #7386

Conversation

yurii-bidiuk commented Oct 26, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

CLAassistant commented Oct 26, 2021 • edited Loading

yurii-bidiuk commented Oct 26, 2021 • edited by github-actions bot Loading

irynakruk left a comment

Choose a reason for hiding this comment

DoNotPanicUA left a comment

Choose a reason for hiding this comment

DoNotPanicUA Oct 26, 2021

Choose a reason for hiding this comment

yurii-bidiuk Oct 26, 2021

Choose a reason for hiding this comment

DoNotPanicUA Oct 26, 2021

Choose a reason for hiding this comment

yurii-bidiuk Oct 26, 2021

Choose a reason for hiding this comment

yurii-bidiuk commented Oct 28, 2021 • edited by github-actions bot Loading

alexandr-shegeda left a comment

Choose a reason for hiding this comment

yurii-bidiuk commented Nov 4, 2021 • edited by github-actions bot Loading

yurii-bidiuk commented Nov 4, 2021 • edited by github-actions bot Loading

yurii-bidiuk commented Nov 4, 2021 • edited by github-actions bot Loading

yurii-bidiuk commented Nov 4, 2021 • edited by github-actions bot Loading

yurii-bidiuk commented Nov 4, 2021 • edited by github-actions bot Loading

yurii-bidiuk commented Oct 26, 2021 •

edited

Loading

CLAassistant commented Oct 26, 2021 •

edited

Loading

yurii-bidiuk commented Oct 26, 2021 •

edited by github-actions bot

Loading

yurii-bidiuk commented Oct 28, 2021 •

edited by github-actions bot

Loading

yurii-bidiuk commented Nov 4, 2021 •

edited by github-actions bot

Loading

yurii-bidiuk commented Nov 4, 2021 •

edited by github-actions bot

Loading

yurii-bidiuk commented Nov 4, 2021 •

edited by github-actions bot

Loading

yurii-bidiuk commented Nov 4, 2021 •

edited by github-actions bot

Loading

yurii-bidiuk commented Nov 4, 2021 •

edited by github-actions bot

Loading