-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-45139][SQL] Add DatabricksDialect to handle SQL type conversion #42896
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
:)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM. Although I didn't test PR, I believe this PR works perfectly. :)
Merged to master for Apache Spark 4.0. |
Thank you @dongjoon-hyun! I forgot to mention in the PR, I did test the dialect manually for a few tables and queries. For example, this query works:
while it fails without the dialect with some type conversion errors. |
Thank you! |
What changes were proposed in this pull request?
This PR adds
DatabricksDialect
to Spark to allow users to query Databricks clusters and Databricks SQL warehouses with more precise SQL type conversion and quote identifiers instead of doing it manually in the code.Why are the changes needed?
The PR fixes type conversion and makes it easier to query Databricks clusters.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
I added unit tests in JDBCSuite to check conversion.
Was this patch authored or co-authored using generative AI tooling?
No.