Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom convert incoming long for date and datetime types #36263

Conversation

rodireich
Copy link
Contributor

@rodireich rodireich commented Mar 19, 2024

Date types included in tables as default values are discovered in debezium with an input of either epoch milliseconds (for Datetime types) or epoch days (for date type).
Our custom converter couldn't convert this input.
Added correct conversion for schema history phase.

Actual record Dates are still seen as an incoming TimeStamp.

Copy link

vercel bot commented Mar 19, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 19, 2024 10:19pm

@octavia-squidington-iii octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/source/mssql labels Mar 19, 2024
@octavia-squidington-iii octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Mar 19, 2024
@rodireich rodireich requested a review from a team March 19, 2024 15:17
@rodireich rodireich changed the title custom convert incoming long for date and datetime types Custom convert incoming long for date and datetime types Mar 19, 2024
@rodireich rodireich marked this pull request as ready for review March 19, 2024 15:39
@rodireich rodireich requested a review from a team as a code owner March 19, 2024 15:39
return localDateTime.format(DateTimeFormatter.ofPattern(DATETIME_FORMAT_MICROSECONDS));
}

if (input instanceof final Long d) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it the case that for Mssql that all timestamp 'numbers' are always with nanosecond precision?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're only seeing this Long input when values are converted as part of acquiring schema history.
Dates on actual row records are still seen as incoming TimeStamp. None of this is documented anywhere as far as I can tell.

We need to take a close look at some point as this area is uneven in terms of implementation.
And some of the code here is almost three years old.

@rodireich
Copy link
Contributor Author

rodireich commented Mar 19, 2024

/publish-java-cdk dry-run=true

🕑 https://github.com/airbytehq/airbyte/actions/runs/8350436389
✅ Successfully published Java CDK version=0.26.0!

…ng-mssql-first-sync-long-cannot-be-cast-to-class-javasqltimestamp
@rodireich
Copy link
Contributor Author

rodireich commented Mar 19, 2024

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/8350519261
✅ Successfully published Java CDK version=0.26.0!

@rodireich rodireich merged commit e5aea95 into master Mar 19, 2024
29 checks passed
@rodireich rodireich deleted the 35946-failure-in-date-time-conversion-during-mssql-first-sync-long-cannot-be-cast-to-class-javasqltimestamp branch March 19, 2024 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit connectors/source/mssql
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failure in date time conversion during mssql first sync: Long cannot be cast to class java.sql.Timestamp
4 participants