[SPARK-32420][SQL] Add handling for unique key in non-codegen hash join

[c21]

What changes were proposed in this pull request?

HashRelation has two separate code paths for unique key look up and non-unique key look up E.g. in its subclass UnsafeHashedRelation, unique key look up is more efficient as it does not have e.g. extra Iterator[UnsafeRow].hasNext()/next() overhead per row.

BroadcastHashJoinExec has handled unique key vs non-unique key separately in code-gen path. But the non-codegen path for broadcast hash join and shuffled hash join do not separate it yet, so adding the support here.

Why are the changes needed?

Shuffled hash join and non-codegen broadcast hash join still rely on this code path for execution. So this PR will help save CPU for executing this two type of join. Adding codegen for shuffled hash join would be a different topic and I will add it in https://issues.apache.org/jira/browse/SPARK-32421 .

Ran the same query as JoinBenchmark, with enabling and disabling this feature. Verified 20% wall clock time improvement (switch control and test group order as well to verify the improvement to not be the noise).

Running benchmark: shuffle hash join
  Running case: shuffle hash join unique key SHJ off
  Stopped after 5 iterations, 4039 ms
  Running case: shuffle hash join unique key SHJ on
  Stopped after 5 iterations, 2898 ms

Java HotSpot(TM) 64-Bit Server VM 1.8.0_181-b13 on Mac OS X 10.15.4
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
shuffle hash join:                        Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
shuffle hash join unique key SHJ off                707            808          81          5.9         168.6       1.0X
shuffle hash join unique key SHJ on                 547            580          50          7.7         130.4       1.3X

Running benchmark: shuffle hash join
  Running case: shuffle hash join unique key SHJ on
  Stopped after 5 iterations, 3333 ms
  Running case: shuffle hash join unique key SHJ off
  Stopped after 5 iterations, 4268 ms

Java HotSpot(TM) 64-Bit Server VM 1.8.0_181-b13 on Mac OS X 10.15.4
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
shuffle hash join:                        Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
shuffle hash join unique key SHJ on                 565            667          60          7.4         134.8       1.0X
shuffle hash join unique key SHJ off                774            854          85          5.4         184.4       0.7X

Does this PR introduce any user-facing change?

No.

How was this patch tested?

• Added test in OuterJoinSuite to cover left outer and right outer join.
• Added test in ExistenceJoinSuite to cover left semi join, and existence join.
• Existing joinSuite already covered inner join.
• Existing ExistenceJoinSuite already covered left anti join, and existence join.

[SparkQA]

Test build #126467 has finished for PR 29216 at commit 09d5a0d.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[c21]

retest this please

[SparkQA]

Test build #126501 has finished for PR 29216 at commit 09d5a0d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[c21]

cc @cloud-fan and @sameeragarwal if you guys can help take a look. Thanks!

[viirya]

Do we already have test for inner join?

[c21]

@viirya - yes, Existing joinSuite already covered inner join.

[viirya]

And test for existenceJoin?

[c21]

@viirya - existence join is tricky and I will try to add one. But besides testing, wondering what do you think of this PR? Thanks.

[c21]

@viirya - I just found test single unique condition (equal) for left Anti join in ExistenceJoinSuite already covered existence join. So we should be good with existence join. More historical context for existence join is this PR just FYI.

[c21]

@viirya - given tests already cover all joins, could you give a review of core logic to help move forward? Thanks.

[cloud-fan]

thanks, merging to master!

[c21]

Thanks @cloud-fan and @viirya for review!