Skip to content
Permalink
Browse files

[SPARK-24007][SQL] EqualNullSafe for FloatType and DoubleType might g…

…enerate a wrong result by codegen.

## What changes were proposed in this pull request?

`EqualNullSafe` for `FloatType` and `DoubleType` might generate a wrong result by codegen.

```scala
scala> val df = Seq((Some(-1.0d), None), (None, Some(-1.0d))).toDF()
df: org.apache.spark.sql.DataFrame = [_1: double, _2: double]

scala> df.show()
+----+----+
|  _1|  _2|
+----+----+
|-1.0|null|
|null|-1.0|
+----+----+

scala> df.filter("_1 <=> _2").show()
+----+----+
|  _1|  _2|
+----+----+
|-1.0|null|
|null|-1.0|
+----+----+
```

The result should be empty but the result remains two rows.

## How was this patch tested?

Added a test.

Author: Takuya UESHIN <ueshin@databricks.com>

Closes #21094 from ueshin/issues/SPARK-24007/equalnullsafe.

(cherry picked from commit f09a9e9)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
  • Loading branch information...
ueshin authored and gatorsmile committed Apr 18, 2018
1 parent 6b99d5b commit a1c56b66970a683e458e3f44fd6788110e869093
@@ -764,8 +764,10 @@ class CodegenContext {
*/
def genEqual(dataType: DataType, c1: String, c2: String): String = dataType match {
case BinaryType => s"java.util.Arrays.equals($c1, $c2)"
case FloatType => s"(java.lang.Float.isNaN($c1) && java.lang.Float.isNaN($c2)) || $c1 == $c2"
case DoubleType => s"(java.lang.Double.isNaN($c1) && java.lang.Double.isNaN($c2)) || $c1 == $c2"
case FloatType =>
s"((java.lang.Float.isNaN($c1) && java.lang.Float.isNaN($c2)) || $c1 == $c2)"
case DoubleType =>
s"((java.lang.Double.isNaN($c1) && java.lang.Double.isNaN($c2)) || $c1 == $c2)"
case dt: DataType if isPrimitiveType(dt) => s"$c1 == $c2"
case dt: DataType if dt.isInstanceOf[AtomicType] => s"$c1.equals($c2)"
case array: ArrayType => genComp(array, c1, c2) + " == 0"
@@ -442,4 +442,11 @@ class PredicateSuite extends SparkFunSuite with ExpressionEvalHelper {
InSet(Literal(1), Set(1, 2, 3, 4)).genCode(ctx)
assert(ctx.inlinedMutableStates.isEmpty)
}

test("SPARK-24007: EqualNullSafe for FloatType and DoubleType might generate a wrong result") {
checkEvaluation(EqualNullSafe(Literal(null, FloatType), Literal(-1.0f)), false)
checkEvaluation(EqualNullSafe(Literal(-1.0f), Literal(null, FloatType)), false)
checkEvaluation(EqualNullSafe(Literal(null, DoubleType), Literal(-1.0d)), false)
checkEvaluation(EqualNullSafe(Literal(-1.0d), Literal(null, DoubleType)), false)
}
}

0 comments on commit a1c56b6

Please sign in to comment.
You can’t perform that action at this time.