[SPARK-37316][SQL] Add code-gen for existence sort merge join

[c21]

What changes were proposed in this pull request?

This PR is to add code-gen for Existence sort merge join. Followed the same algorithm used in iterator mode from SortMergeJoinExec.scala and HashJoin.scala. Check every left side row if there's an existing match from right side. Output every left side row and boolean flag to indicate whether it has a match or not.

In addition, to help review as this PR triggers several TPCDS plan files change. The below file is having the real code change:

• SortMergeJoinExec.scala

Why are the changes needed?

To improve performance for Existence sort merge join, and conclude all join types code-gen support for sort merge join.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing unit test in ExistenceJoinSuite.scala

[c21]

Just FYI - this is using same logic as hash join - https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala#L667-L668 .

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49702/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49707/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49702/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49707/

[SparkQA]

Test build #145232 has finished for PR 34601 at commit 95e6c0f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #145237 has finished for PR 34601 at commit 77a386c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49717/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49717/

[c21]

cc @cloud-fan could you help take a look when you have time? Thanks.

[SparkQA]

Test build #145247 has finished for PR 34601 at commit f7906be.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!

[c21]

Thank you @cloud-fan for review!