-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-28152][DOCS][FOLLOWUP] Add a migration guide for MsSQLServer JDBC dialect #27270
Conversation
cc @gatorsmile and @viirya |
Test build #116966 has finished for PR 27270 at commit
|
@@ -344,6 +344,12 @@ license: | | |||
|
|||
- Since Spark 2.4.5, `TRUNCATE TABLE` command tries to set back original permission and ACLs during re-creating the table/partition paths. To restore the behaviour of earlier versions, set `spark.sql.truncateTable.ignorePermissionAcl.enabled` to `true`. | |||
|
|||
- Since Spark 2.4.5, `spark.sql.legacy.mssqlserver.numericMapping.enabled` configuration is added in order to support the legacy MsSQLServer dialect mapping behavior using IntegerType and DoubleType for SMALLINT and REAL JDBC types, respectively. To restore the behaviour of 2.4.3 and earlier versions, set `spark.sql.legacy.mssqlserver.numericMapping.enabled` to `true`. | |||
|
|||
## Upgrading from Spark SQL 2.4.3 to 2.4.4 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As 2.4.4 is already out, this doc is in 2.4.5? So maybe it should be 2.4.3 -> 2.4.5?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO, although it's late for 2.4.4, 2.4.3
to 2.4.4
will be correct. When the users upgrade from 1.6.3 to 3.0.0, they need to see all previous migration guides. If there is some regression on 2.4.5, the users can use 2.4.4 instead of 2.4.5.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For 2.4.4
release doc, I'll update spark-website
repository.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok sounds good. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
Merged to master/2.4. |
Thank you, @maropu ! |
…DBC dialect This PR adds a migration guide for MsSQLServer JDBC dialect for Apache Spark 2.4.4 and 2.4.5. Apache Spark 2.4.4 updates the type mapping correctly according to MS SQL Server, but missed to mention that in the migration guide. In addition, 2.4.4 adds a configuration for the legacy behavior. Yes. This is a documentation change. ![screenshot](https://user-images.githubusercontent.com/9700541/72649944-d6517780-3933-11ea-92be-9d4bf38e2eda.png) Manually generate and see the doc. Closes #27270 from dongjoon-hyun/SPARK-28152-DOC. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 505693c) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
hahaha, I was a bit late ;) |
You are faster than me. :) |
What changes were proposed in this pull request?
This PR adds a migration guide for MsSQLServer JDBC dialect for Apache Spark 2.4.4 and 2.4.5.
Why are the changes needed?
Apache Spark 2.4.4 updates the type mapping correctly according to MS SQL Server, but missed to mention that in the migration guide. In addition, 2.4.4 adds a configuration for the legacy behavior.
Does this PR introduce any user-facing change?
Yes. This is a documentation change.
How was this patch tested?
Manually generate and see the doc.