Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Spark-compatible CAST from String to Timestamp #328

Closed
andygrove opened this issue Apr 25, 2024 · 2 comments · Fixed by #335 or #337
Closed

Implement Spark-compatible CAST from String to Timestamp #328

andygrove opened this issue Apr 25, 2024 · 2 comments · Fixed by #335 or #337
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@andygrove
Copy link
Member

What is the problem the feature request solves?

We currently delegate to DataFusion when casting from string to timestamp and there are some differences in behavior compared to Spark.

  • Spark treats many parts of the timestamp as optional
  • There are timezone-related differences
  • DataFusion does not throw exceptions on invalid inputs in ANSI mode

Examples of differences:

Input Spark Comet
2020 2020-01-01 00:00:00.0 null
2020-01 2020-01-01 00:00:00.0 null
2020-01-01 2020-01-01 00:00:00.0 2019-12-31 16:00:00.0
2020-01-01T12 2020-01-01 12:00:00.0 null
2020-01-01T12:34 2020-01-01 12:34:00.0 null
2020-01-01T12:34:56 2020-01-01 12:34:56.0 2020-01-01 04:34:56.0
2020-01-01T12:34:56.123456 2020-01-01 12:34:56.123456 2020-01-01 04:34:56.123456
T2 2024-04-25 02:00:00.0 null

Describe the potential solution

No response

Additional context

No response

@andygrove andygrove added the enhancement New feature or request label Apr 25, 2024
@andygrove andygrove added good first issue Good for newcomers help wanted Extra attention is needed labels Apr 25, 2024
@vaibhawvipul
Copy link
Contributor

vaibhawvipul commented Apr 26, 2024

I would love to work on this. If no one else is working on it.

@andygrove
Copy link
Member Author

There is a follow on issue to complete this work: #376

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
2 participants