Skip to content

Commit

Permalink
[SPARK-30339][SQL][2.4] Avoid to fail twice in function lookup
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

Backported from [pr#26994](#26994). Currently if function lookup fails, spark will give it a second chance by casting decimal type to double type. But for cases where decimal type doesn't exist, it's meaningless to lookup again and cause extra cost like unnecessary metastore access. We should throw exceptions directly in these cases.

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?

Covered by existing tests.

Closes #27054 from wzhfy/avoid_udf_fail_twice-2.4.

Authored-by: Zhenhua Wang <wzh_zju@163.com>
Signed-off-by: Zhenhua Wang <wzh_zju@163.com>
  • Loading branch information
wzhfy committed Dec 31, 2019
1 parent db32408 commit 03cea11
Show file tree
Hide file tree
Showing 2 changed files with 1 addition and 8 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ private[sql] class HiveSessionCatalog(
try {
lookupFunction0(name, children)
} catch {
case NonFatal(_) =>
case NonFatal(_) if children.exists(_.dataType.isInstanceOf[DecimalType]) =>
// SPARK-16228 ExternalCatalog may recognize `double`-type only.
val newChildren = children.map { child =>
if (child.dataType.isInstanceOf[DecimalType]) Cast(child, DoubleType) else child
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -148,13 +148,6 @@ class HiveUDFSuite extends QueryTest with TestHiveSingleton with SQLTestUtils {
sql("SELECT array(max(key), max(key)) FROM src").collect().toSeq)
}

test("SPARK-16228 Percentile needs explicit cast to double") {
sql("select percentile(value, cast(0.5 as double)) from values 1,2,3 T(value)")
sql("select percentile_approx(value, cast(0.5 as double)) from values 1.0,2.0,3.0 T(value)")
sql("select percentile(value, 0.5) from values 1,2,3 T(value)")
sql("select percentile_approx(value, 0.5) from values 1.0,2.0,3.0 T(value)")
}

test("Generic UDAF aggregates") {

checkAnswer(sql(
Expand Down

0 comments on commit 03cea11

Please sign in to comment.