-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-30339][SQL] Avoid to fail twice in function lookup #26994
Conversation
Test build #115679 has finished for PR 26994 at commit
|
retest this please |
Test build #115688 has finished for PR 26994 at commit
|
retest this please |
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala
Show resolved
Hide resolved
Test build #115753 has finished for PR 26994 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you remove
spark/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
Lines 152 to 157 in 9a3dcca
test("SPARK-16228 Percentile needs explicit cast to double") { | |
sql("select percentile(value, cast(0.5 as double)) from values 1,2,3 T(value)") | |
sql("select percentile_approx(value, cast(0.5 as double)) from values 1.0,2.0,3.0 T(value)") | |
sql("select percentile(value, 0.5) from values 1,2,3 T(value)") | |
sql("select percentile_approx(value, 0.5) from values 1.0,2.0,3.0 T(value)") | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks fine with @maropu's suggestion.
Test build #115955 has finished for PR 26994 at commit
|
Test build #115957 has finished for PR 26994 at commit
|
Merged to master. |
### What changes were proposed in this pull request? Currently if function lookup fails, spark will give it a second change by casting decimal type to double type. But for cases where decimal type doesn't exist, it's meaningless to lookup again and causes extra cost like unnecessary metastore access. We should throw exceptions directly in these cases. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Covered by existing tests. Closes apache#26994 from wzhfy/avoid_udf_fail_twice. Authored-by: Zhenhua Wang <wzh_zju@163.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
### What changes were proposed in this pull request? Backported from [pr#26994](#26994). Currently if function lookup fails, spark will give it a second chance by casting decimal type to double type. But for cases where decimal type doesn't exist, it's meaningless to lookup again and cause extra cost like unnecessary metastore access. We should throw exceptions directly in these cases. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Covered by existing tests. Closes #27054 from wzhfy/avoid_udf_fail_twice-2.4. Authored-by: Zhenhua Wang <wzh_zju@163.com> Signed-off-by: Zhenhua Wang <wzh_zju@163.com>
Late LGTM. Thanks, @HyukjinKwon ! |
What changes were proposed in this pull request?
Currently if function lookup fails, spark will give it a second chance by casting decimal type to double type. But for cases where decimal type doesn't exist, it's meaningless to lookup again and cause extra cost like unnecessary metastore access. We should throw exceptions directly in these cases.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Covered by existing tests.