[SPARK-24151][SQL] Case insensitive resolution of CURRENT_DATE and CURRENT_TIMESTAMP #22440
Conversation
docs/sql-programming-guide.md
Outdated
| @@ -1879,6 +1879,7 @@ working with timestamps in `pandas_udf`s to get the best performance, see | |||
|
|
|||
| ## Upgrading From Spark SQL 2.3 to 2.4 | |||
|
|
|||
| - In version 2.3 and earlier, if `spark.sql.caseSensitive` is set to true, then the `CURRENT_DATE` and `CURRENT_TIMESTAMP` functions incorrectly became case-sensitive and would resolve to columns (unless typed in lower case). In Spark 2.4 this has been fixed and the functions are no longer case-sensitive. | |||
There was a problem hiding this comment.
Shall we give the specified version range like https://github.com/apache/spark/pull/21217/files#r203205223?
There was a problem hiding this comment.
I updated specifying the versions. As this is included in the migration guide to 2.4 and I don't think this commit will be backported to 2.3, I think it is fine to state 2.3, but it is good to mention that also 2.2.1 is affected. Thanks.
|
ok to test |
|
Test build #96130 has finished for PR 22440 at commit
|
|
Thank you, @mgaido91 . |
|
Test build #96136 has finished for PR 22440 at commit
|
|
@mgaido91 . I know that your test code is bigger than the remaining original code in However, we changes our way of credits into an explicit manner. If you preserve some of @jamesthomp 's commits, he will be the original author and you will be a co-author as you requested. I prefer this way. Otherwise, could you add an empty commit with @jamesthomp 's name and email at least? In that case, he will be co-author automatically. |
|
Test build #96129 has finished for PR 22440 at commit
|
|
I see @dongjoon-hyun. Sure, sorry I wasn't aware of how this is working. Shall I create a new PR based on @jamesthomp's commits or is it enough to update this adding his commits? Sorry for the trouble and thanks for guiding me on this. |
|
For now, I think it's enough to add an empty commit for him in this PR. Then, I will try to moderate during setting the main author. |
|
Thanks! Merged to master/2.4 |
…RRENT_TIMESTAMP ## What changes were proposed in this pull request? SPARK-22333 introduced a regression in the resolution of `CURRENT_DATE` and `CURRENT_TIMESTAMP`. Before that ticket, these 2 functions were resolved in a case insensitive way. After, this depends on the value of `spark.sql.caseSensitive`. The PR restores the previous behavior and makes their resolution case insensitive anyhow. The PR takes over #21217, therefore it closes #21217 and credit for this patch should be given to jamesthomp. ## How was this patch tested? added UT Closes #22440 from mgaido91/SPARK-24151. Lead-authored-by: James Thompson <jamesthomp@users.noreply.github.com> Co-authored-by: Marco Gaido <marcogaido91@gmail.com> Signed-off-by: gatorsmile <gatorsmile@gmail.com> (cherry picked from commit ba838fe) Signed-off-by: gatorsmile <gatorsmile@gmail.com>
|
Thank you for merging, @gatorsmile ! I like it. |
What changes were proposed in this pull request?
SPARK-22333 introduced a regression in the resolution of
CURRENT_DATEandCURRENT_TIMESTAMP. Before that ticket, these 2 functions were resolved in a case insensitive way. After, this depends on the value ofspark.sql.caseSensitive.The PR restores the previous behavior and makes their resolution case insensitive anyhow. The PR takes over #21217, therefore it closes #21217 and credit for this patch should be given to @jamesthomp.
How was this patch tested?
added UT