Skip to content

Commit

Permalink
[SPARK-40315][SQL] Add hashCode() for Literal of ArrayBasedMapData
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
There is no explicit `hashCode()` function override for `ArrayBasedMapData`. As a result, there is a non-deterministic error where the `hashCode()` computed for `Literal`s of `ArrayBasedMapData` can be different for two equal objects (`Literal`s of `ArrayBasedMapData` with equal keys and values).

In this PR, we add a `hashCode` function so that it works exactly as we expect.

### Why are the changes needed?
This is a bug fix for a non-deterministic error. It is also more consistent with the rest of Spark if we implement the `hashCode` method instead of relying on defaults. We can't add the `hashCode` directly to `ArrayBasedMapData` because of SPARK-9415.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
A simple unit test was added.

Closes #37807 from c27kwan/SPARK-40315-lit.

Authored-by: Carmen Kwan <carmen.kwan@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit e85a4ff)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
c27kwan authored and cloud-fan committed Sep 6, 2022
1 parent 1324f7d commit 84c0918
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -374,6 +374,9 @@ case class Literal (value: Any, dataType: DataType) extends LeafExpression {
val valueHashCode = value match {
case null => 0
case binary: Array[Byte] => util.Arrays.hashCode(binary)
// SPARK-40315: Literals of ArrayBasedMapData should have deterministic hashCode.
case arrayBasedMapData: ArrayBasedMapData =>
arrayBasedMapData.keyArray.hashCode() * 37 + arrayBasedMapData.valueArray.hashCode()
case other => other.hashCode()
}
31 * Objects.hashCode(dataType) + valueHashCode
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -526,4 +526,30 @@ class ComplexTypeSuite extends SparkFunSuite with ExpressionEvalHelper {

assert(m1.semanticEquals(m2))
}

test("SPARK-40315: Literals of ArrayBasedMapData should have deterministic hashCode.") {
val keys = new Array[UTF8String](1)
val values1 = new Array[UTF8String](1)
val values2 = new Array[UTF8String](1)

keys(0) = UTF8String.fromString("key")
values1(0) = UTF8String.fromString("value1")
values2(0) = UTF8String.fromString("value2")

val d1 = new ArrayBasedMapData(new GenericArrayData(keys), new GenericArrayData(values1))
val d2 = new ArrayBasedMapData(new GenericArrayData(keys), new GenericArrayData(values1))
val d3 = new ArrayBasedMapData(new GenericArrayData(keys), new GenericArrayData(values2))
val m1 = Literal.create(d1, MapType(StringType, StringType))
val m2 = Literal.create(d2, MapType(StringType, StringType))
val m3 = Literal.create(d3, MapType(StringType, StringType))

// If two Literals of ArrayBasedMapData have the same elements, we expect them to be equal and
// to have the same hashCode().
assert(m1 == m2)
assert(m1.hashCode() == m2.hashCode())
// If two Literals of ArrayBasedMapData have different elements, we expect them not to be equal
// and to have different hashCode().
assert(m1 != m3)
assert(m1.hashCode() != m3.hashCode())
}
}

0 comments on commit 84c0918

Please sign in to comment.