Skip to content

Commit

Permalink
[SPARK-23938][SQL][FOLLOW-UP][TEST] Nullabilities of value arguments …
Browse files Browse the repository at this point in the history
…should be true.

## What changes were proposed in this pull request?

This is a follow-up pr of #22017 which added `map_zip_with` function.
In the test, when creating a lambda function, we use the `valueContainsNull` values for the nullabilities of the value arguments, but we should've used `true` as the same as `bind` method because the values might be `null` if the keys don't match.

## How was this patch tested?

Added small tests and existing tests.

Closes #22126 from ueshin/issues/SPARK-23938/fix_tests.

Authored-by: Takuya UESHIN <ueshin@databricks.com>
Signed-off-by: Takuya UESHIN <ueshin@databricks.com>
  • Loading branch information
ueshin committed Aug 17, 2018
1 parent 8af61fb commit c1ffb3c
Showing 1 changed file with 7 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -436,9 +436,9 @@ class HigherOrderFunctionsSuite extends SparkFunSuite with ExpressionEvalHelper
left: Expression,
right: Expression,
f: (Expression, Expression, Expression) => Expression): Expression = {
val MapType(kt, vt1, vcn1) = left.dataType.asInstanceOf[MapType]
val MapType(_, vt2, vcn2) = right.dataType.asInstanceOf[MapType]
MapZipWith(left, right, createLambda(kt, false, vt1, vcn1, vt2, vcn2, f))
val MapType(kt, vt1, _) = left.dataType
val MapType(_, vt2, _) = right.dataType
MapZipWith(left, right, createLambda(kt, false, vt1, true, vt2, true, f))
}

val mii0 = Literal.create(Map(1 -> 10, 2 -> 20, 3 -> 30),
Expand Down Expand Up @@ -475,6 +475,8 @@ class HigherOrderFunctionsSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(
map_zip_with(mii0, miin, multiplyKeyWithValues),
null)
assert(map_zip_with(mii0, mii1, multiplyKeyWithValues).dataType ===
MapType(IntegerType, IntegerType, valueContainsNull = true))

val mss0 = Literal.create(Map("a" -> "x", "b" -> "y", "d" -> "z"),
MapType(StringType, StringType, valueContainsNull = false))
Expand Down Expand Up @@ -510,6 +512,8 @@ class HigherOrderFunctionsSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(
map_zip_with(mss0, mssn, concat),
null)
assert(map_zip_with(mss0, mss1, concat).dataType ===
MapType(StringType, StringType, valueContainsNull = true))

def b(data: Byte*): Array[Byte] = Array[Byte](data: _*)

Expand Down

0 comments on commit c1ffb3c

Please sign in to comment.