[Source-Mssql] Format Datetime and Datetime2 datatypes to 6-digit microsecond precision #32573

nguyenaiden · 2023-11-15T20:23:26Z

Description: SQLServer's datetime2 data type has a max timestamp precision of 7, but for some destinations, specifically BigQuery, sending a record with this precision - e.g: 2023-11-08T01:20:11.3733338 - will be rejected by BigQuery with a thrown error, and this leads to a dropped record on the other end as the entry will show up as null
This also updated datetime to have the same precision for consistency.

Temporary solution is to enforce a 6-digit precision after reading the record from the source before data emission.

Mssql Record	Airbyte Record
9999-12-31T13:00:04.12345	9999-12-31T13:00:04.123450
2023-11-08T01:20:11.3733338	2023-11-08T01:20:11.373333

Note: This is to temporarily unblock a potential customer. This will be reverted when Destination/Platform start to handle this issue on their end.

vercel · 2023-11-15T20:23:31Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Nov 30, 2023 7:56pm

github-actions · 2023-11-15T20:23:45Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

stephane-airbyte

just one small comment

stephane-airbyte · 2023-11-15T20:52:33Z

...s/source-mssql/src/main/java/io/airbyte/integrations/source/mssql/MssqlSourceOperations.java

+    try {
+      node.put(columnName, getObject(resultSet, index, LocalDateTime.class).format(microsecondsFormatter));
+    } catch (final Exception e) {
+      // for backward compatibility


what does that mean? Should we at least log something here ?

After thinking about it, I'm going to remove this section altogether. This was done inside of AbstractJdbcCompatibleSourceOperations#putTimestamp for generic JDBC sources because we didn't know if we were able to parse the result to the appropriate Datetime/Date/Timestamp Java type. But since we know how a MSSQL datetime result looks like, this isn't necessary.

rodireich · 2023-11-30T17:24:13Z

...urces/src/main/java/io/airbyte/cdk/integrations/debezium/internals/mssql/MSSQLConverter.java

  private final Set<String> BINARY = Set.of("VARBINARY", "BINARY");
+  private final Set<String> DATETIME_TYPES = Set.of("DATETIME", "DATETIME2", "SMALLDATETIME");


I see also date, datetimeoffset, time.
Any other one also relevant?

Those 3 are the only ones that gets translated into timestamp. Datetimeoffset is timestamp with timezone, and the remaining date and time have their own translation.

rodireich · 2023-11-30T17:25:18Z

...urces/src/main/java/io/airbyte/cdk/integrations/debezium/internals/mssql/MSSQLConverter.java

  private static final String DATETIMEOFFSET = "DATETIMEOFFSET";
  private static final String TIME_TYPE = "TIME";
  private static final String SMALLMONEY_TYPE = "SMALLMONEY";
  private static final String GEOMETRY = "GEOMETRY";
  private static final String GEOGRAPHY = "GEOGRAPHY";
  private static final String DEBEZIUM_DATETIMEOFFSET_FORMAT = "yyyy-MM-dd HH:mm:ss XXX";

+  private static final String DATETIME_FORMAT_MICROSECONDS = "yyyy-MM-dd'T'HH:mm:ss[.][SSSSSS]";


We're hard coding it to 6 to fit with most of the popular destinations for now. This work is slated to be reverted once Destination/platform establish a precision standard on their end.

rodireich · 2023-11-30T17:26:31Z

airbyte-integrations/connectors/source-mssql/build.gradle

@@ -6,7 +6,7 @@ plugins {
 airbyteJavaConnector {
    cdkVersionRequired = '0.5.0'
    features = ['db-sources']
-    useLocalCdk = false
+    useLocalCdk = true


ℹ️ Reminder to change before merging

…ond precision

…ow how MSSQL datetime results look like

nguyenaiden · 2023-11-30T19:17:09Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/7051227850
✅ Successfully published Java CDK version=0.6.2!

…rict-encrypt

octavia-squidington-iii added area/connectors Connector related issues connectors/source/mssql connectors/source/mssql-strict-encrypt labels Nov 15, 2023

nguyenaiden force-pushed the mssql-datetime-patch branch from 0d4cd18 to 7fe6c3d Compare November 15, 2023 20:26

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Nov 15, 2023

vercel bot deployed to Preview November 15, 2023 20:28 View deployment

nguyenaiden marked this pull request as ready for review November 15, 2023 20:37

nguyenaiden requested a review from a team as a code owner November 15, 2023 20:37

nguyenaiden changed the title ~~[Source-Mssql] Format Datetime and Datetime2 datatypes to 6 digit Microsecond precision~~ [Source-Mssql] Format Datetime and Datetime2 datatypes to 6-digit microsecond precision Nov 15, 2023

nguyenaiden requested review from rodireich, postamar, xiaohansong, akashkulk, prateekmukhedkar and stephane-airbyte November 15, 2023 20:39

stephane-airbyte approved these changes Nov 15, 2023

View reviewed changes

nguyenaiden force-pushed the mssql-datetime-patch branch from 9b0dc29 to 73449e8 Compare November 15, 2023 21:10

nguyenaiden force-pushed the mssql-datetime-patch branch from 2007a67 to f61375e Compare November 30, 2023 00:13

octavia-squidington-iii added the CDK Connector Development Kit label Nov 30, 2023

nguyenaiden requested a review from stephane-airbyte November 30, 2023 00:13

rodireich reviewed Nov 30, 2023

View reviewed changes

rodireich approved these changes Nov 30, 2023

View reviewed changes

nguyenaiden force-pushed the mssql-datetime-patch branch 2 times, most recently from adc9e2b to ddcddc7 Compare November 30, 2023 18:43

nguyenaiden added 4 commits November 30, 2023 11:13

Format datetime and datetime2 records to always have 6 digit microsec…

a720677

…ond precision

Updated mssql.md

219d424

Remove backwards compatibility caught exception section because we kn…

2de6929

…ow how MSSQL datetime results look like

Update precision for CDC as well

d07de3a

nguyenaiden added 3 commits November 30, 2023 11:14

Rebased

4a66b94

USe local cdk for strict encrypt temporarily

cfe38c8

Bumped cdk to 0.6.2

e1eebd9

nguyenaiden force-pushed the mssql-datetime-patch branch from eaad3d8 to e1eebd9 Compare November 30, 2023 19:15

Bumped CDK version and revert useLocalCdk flag for mssql and mssql-st…

83106c6

…rict-encrypt

nguyenaiden force-pushed the mssql-datetime-patch branch from d56b5fc to 83106c6 Compare November 30, 2023 19:52

Automated Commit - Formatting Changes

e4b8153

nguyenaiden merged commit 5acca55 into master Nov 30, 2023
24 checks passed

nguyenaiden deleted the mssql-datetime-patch branch November 30, 2023 20:17

rodireich mentioned this pull request Feb 27, 2024

Source MSSQL: Pipeline sync issue for sql server source with CDC update method #34906

Closed

rodireich mentioned this pull request Mar 11, 2024

Failure in date time conversion during mssql first sync: Long cannot be cast to class java.sql.Timestamp #35946

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Source-Mssql] Format Datetime and Datetime2 datatypes to 6-digit microsecond precision #32573

[Source-Mssql] Format Datetime and Datetime2 datatypes to 6-digit microsecond precision #32573

nguyenaiden commented Nov 15, 2023 •

edited

Loading

vercel bot commented Nov 15, 2023 •

edited

Loading

github-actions bot commented Nov 15, 2023 •

edited by nguyenaiden

Loading

stephane-airbyte left a comment

stephane-airbyte Nov 15, 2023

nguyenaiden Nov 15, 2023

rodireich Nov 30, 2023

nguyenaiden Nov 30, 2023

rodireich Nov 30, 2023

nguyenaiden Nov 30, 2023

rodireich Nov 30, 2023

nguyenaiden commented Nov 30, 2023 •

edited by github-actions bot

Loading

		private final Set<String> BINARY = Set.of("VARBINARY", "BINARY");
		private final Set<String> DATETIME_TYPES = Set.of("DATETIME", "DATETIME2", "SMALLDATETIME");

[Source-Mssql] Format Datetime and Datetime2 datatypes to 6-digit microsecond precision #32573

[Source-Mssql] Format Datetime and Datetime2 datatypes to 6-digit microsecond precision #32573

Conversation

nguyenaiden commented Nov 15, 2023 • edited Loading

vercel bot commented Nov 15, 2023 • edited Loading

github-actions bot commented Nov 15, 2023 • edited by nguyenaiden Loading

Before Merging a Connector Pull Request

stephane-airbyte left a comment

Choose a reason for hiding this comment

stephane-airbyte Nov 15, 2023

Choose a reason for hiding this comment

nguyenaiden Nov 15, 2023

Choose a reason for hiding this comment

rodireich Nov 30, 2023

Choose a reason for hiding this comment

nguyenaiden Nov 30, 2023

Choose a reason for hiding this comment

rodireich Nov 30, 2023

Choose a reason for hiding this comment

nguyenaiden Nov 30, 2023

Choose a reason for hiding this comment

rodireich Nov 30, 2023

Choose a reason for hiding this comment

nguyenaiden commented Nov 30, 2023 • edited by github-actions bot Loading

nguyenaiden commented Nov 15, 2023 •

edited

Loading

vercel bot commented Nov 15, 2023 •

edited

Loading

github-actions bot commented Nov 15, 2023 •

edited by nguyenaiden

Loading

nguyenaiden commented Nov 30, 2023 •

edited by github-actions bot

Loading