Skip to content

Commit

Permalink
[SPARK-22464][SQL] No pushdown for Hive metastore partition predicate…
Browse files Browse the repository at this point in the history
…s containing null-safe equality

## What changes were proposed in this pull request?
`<=>` is not supported by Hive metastore partition predicate pushdown. We should not push down it to Hive metastore when they are be using in partition predicates.

## How was this patch tested?
Added a test case

Author: gatorsmile <gatorsmile@gmail.com>

Closes #19682 from gatorsmile/fixLimitPushDown.

# Conflicts:
#	sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
#	sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
  • Loading branch information
gatorsmile committed Nov 11, 2017
1 parent 4ef0bef commit ec7e4b8
Showing 1 changed file with 16 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -576,6 +576,18 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
hive.getFunctions(db, pattern).asScala
}

/**
* An extractor that matches all binary comparison operators except null-safe equality.
*
* Null-safe equality is not supported by Hive metastore partition predicate pushdown
*/
object SpecialBinaryComparison {
def unapply(e: BinaryComparison): Option[(Expression, Expression)] = e match {
case _: EqualNullSafe => None
case _ => Some((e.left, e.right))
}
}

/**
* Converts catalyst expression to the format that Hive's getPartitionsByFilter() expects, i.e.
* a string that represents partition predicates like "str_key=\"value\" and int_key=1 ...".
Expand All @@ -590,14 +602,14 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
.map(col => col.getName).toSet

filters.collect {
case op @ BinaryComparison(a: Attribute, Literal(v, _: IntegralType)) =>
case op @ SpecialBinaryComparison(a: Attribute, Literal(v, _: IntegralType)) =>
s"${a.name} ${op.symbol} $v"
case op @ BinaryComparison(Literal(v, _: IntegralType), a: Attribute) =>
case op @ SpecialBinaryComparison(Literal(v, _: IntegralType), a: Attribute) =>
s"$v ${op.symbol} ${a.name}"
case op @ BinaryComparison(a: Attribute, Literal(v, _: StringType))
case op @ SpecialBinaryComparison(a: Attribute, Literal(v, _: StringType))
if !varcharKeys.contains(a.name) =>
s"""${a.name} ${op.symbol} ${quoteStringLiteral(v.toString)}"""
case op @ BinaryComparison(Literal(v, _: StringType), a: Attribute)
case op @ SpecialBinaryComparison(Literal(v, _: StringType), a: Attribute)
if !varcharKeys.contains(a.name) =>
s"""${quoteStringLiteral(v.toString)} ${op.symbol} ${a.name}"""
}.mkString(" and ")
Expand Down

0 comments on commit ec7e4b8

Please sign in to comment.