[SPARK-35221][SQL] Add the check of supported join hints #32355

ulysses-you · 2021-04-27T02:59:45Z

What changes were proposed in this pull request?

Print warning msg if join hint is not supported for the specified build side.

Why are the changes needed?

Currently we support specify the join implementation with hint, but Spark did not promise it.

For example broadcast outer join and hash outer join we need to check if its build side was supported. And at least we should print some warning log instead of changing to other join implementation silently.

Does this PR introduce any user-facing change?

Yes, warning log might be printed.

How was this patch tested?

Add new test.

SparkQA · 2021-04-27T04:19:41Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42498/

SparkQA · 2021-04-27T04:25:19Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42498/

ulysses-you · 2021-04-27T05:54:52Z

thanks for review @maropu @cloud-fan

cloud-fan · 2021-04-27T06:15:09Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HintErrorLogger.scala

@@ -42,6 +45,17 @@ object HintErrorLogger extends HintErrorHandler with Logging {
    logWarning(s"A join hint $hint is specified but it is not part of a join relation.")
  }

+  override def joinBuildSideNotSupported(joinType: JoinType, joinHint: JoinHint): Unit = {


cc @maryannxue

can we make the name more general like hintNotSupported?

e.g.

def hintNotSupported(hint: HintInfo, reason: String): Unit = { logWarning("Hint $hint is not supported in the query: " + reason) }

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala

SparkQA · 2021-04-27T08:10:17Z

Test build #137978 has finished for PR 32355 at commit 97029f2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2021-04-28T08:32:37Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/hints.scala

+  /**
+   * Callback for a join hint specified on a join that doesn't support this build side.
+   */
+  def joinBuildSideNotSupported(joinType: JoinType, joinHint: JoinHint): Unit


nit: how about joinBuildSideNotSupported -> hintBuildSideNotSupported ?

maropu · 2021-04-28T08:33:30Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HintErrorLogger.scala

+    } else {
+      (joinHint.rightHint.get, "right")
+    }
+    logWarning(s"A join hint $hint is specified but it is not supported with build $buildSide " +


nit: A join hint $hint => Hint $hint to follow the existing msg: https://github.com/apache/spark/pull/32355/files#diff-afe669f38210f85c87450ace74311b4ce2301e6aeab820aea444fcd802c2d5c2R60

This msg format and method follow that joinNotFoundForJoinHint msg. Seems that one is better ?

HyukjinKwon · 2021-08-01T02:07:56Z

sql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala

+        }
+
+        val logs = hintAppender.loggingEvents.map(_.getRenderedMessage)
+          .filter(_.startsWith("A join hint"))


nit but I think the test here can be flaky (e.g., the test will be broken if there's another log with "A join hint" added). What about just checking if there's a log that contains the message instead of checking the length of these logs?

thank you @HyukjinKwon , agree. Change to assert(logs.nonEmpty) that ensure logs.forall works.

…5221

SparkQA · 2021-08-02T13:35:09Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46466/

SparkQA · 2021-08-02T14:09:30Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46466/

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala

SparkQA · 2021-08-02T18:05:50Z

Test build #141955 has finished for PR 32355 at commit f231e31.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ulysses-you · 2021-08-03T04:42:25Z

thank you @cloud-fan for review, added two methods in JoinSelection:

checkHintBuildSide is to check hint build side
checkHintNonEquiJoin is to check hint equi join

And also added test for the equi join check.

SparkQA · 2021-08-03T05:34:56Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46492/

cloud-fan · 2021-08-03T07:58:53Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/hints.scala

@@ -213,6 +213,11 @@ trait HintErrorHandler {
   */
  def joinNotFoundForJoinHint(hint: HintInfo): Unit

+  /**
+   * Callback for a join hint specified on a join that doesn't support this build side.


if we only target for join hint, let's name it joinHintNotSupported

SparkQA · 2021-08-03T09:31:51Z

Test build #141981 has finished for PR 32355 at commit 11c5700.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-03T09:34:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46497/

SparkQA · 2021-08-03T10:12:48Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46497/

cloud-fan · 2021-08-03T10:18:13Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala

+      if (hintToShuffleHashJoin(hint) || hintToSortMergeJoin(hint)) {
+        assert(hint.leftHint.orElse(hint.rightHint).isDefined)
+        hintErrorHandler.joinHintNotSupported(hint.leftHint.orElse(hint.rightHint).get,
+          "equi join keys is not existed")


no equi-join keys

sql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala

SparkQA · 2021-08-03T12:35:17Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46506/

SparkQA · 2021-08-03T13:12:08Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46506/

SparkQA · 2021-08-03T16:29:39Z

Test build #141994 has finished for PR 32355 at commit 2bb4200.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-03T17:13:42Z

Test build #141986 has finished for PR 32355 at commit 307c8ac.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds no public classes.

ulysses-you · 2021-08-04T01:29:35Z

retest this please

SparkQA · 2021-08-04T02:44:55Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46524/

SparkQA · 2021-08-04T06:31:42Z

Test build #142012 has finished for PR 32355 at commit 2bb4200.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-08-05T12:00:01Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala

+        hint: JoinHint,
+        isBroadcast: Boolean): Unit = {
+      if (onlyLookingAtHint && buildSide.isEmpty) {
+        if ((isBroadcast && hintToBroadcastLeft(hint)) ||


the code looks a bit messy, we can clearly show the branches:

def invalidBuildSideInHint(buildSide: String) = { hintErrorHandler.joinHintNotSupported(hint.leftHint.get, s"build $buildSide for ${joinType.sql.toLowerCase(Locale.ROOT)} join") } if (onlyLookingAtHint && buildSide.isEmpty) { if (broadcast) { // check broadcast hash join if (hintToBroadcastLeft) invalidBuildSideInHint("left") if (hintToBroadcastRight) invalidBuildSideInHint("right") } else { // check shuffle hash join ... } }

looks better

cloud-fan · 2021-08-05T12:02:58Z

sql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala

+
+        val logs = hintAppender.loggingEvents.map(_.getRenderedMessage)
+          .filter(_.contains("is not supported in the query:"))
+        assert(logs.nonEmpty)


shall we assert logs.length == 2?

It should be. According to the #32355 (comment) and to make it clear. Now I split the related test into parties: positive and negative. I think it would be less flaky FYI @HyukjinKwon .

cloud-fan · 2021-08-05T12:03:06Z

sql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala

+        }
+        val logs = hintAppender.loggingEvents.map(_.getRenderedMessage)
+          .filter(_.contains("is not supported in the query:"))
+        assert(logs.nonEmpty)


SparkQA · 2021-08-05T15:07:43Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46610/

SparkQA · 2021-08-05T15:45:23Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46610/

SparkQA · 2021-08-05T15:49:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46612/

SparkQA · 2021-08-05T16:45:16Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46612/

cloud-fan · 2021-08-05T17:05:25Z

sql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala

+    }
+    val logs = hintAppender.loggingEvents.map(_.getRenderedMessage)
+      .filter(_.contains("is not supported in the query:"))
+    assert(logs.nonEmpty)


should this be size 2 as well?

SparkQA · 2021-08-05T18:58:21Z

Test build #142098 has finished for PR 32355 at commit d90dd66.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-05T19:37:11Z

Test build #142100 has finished for PR 32355 at commit 5ac7952.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-06T00:44:22Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46628/

SparkQA · 2021-08-06T04:30:44Z

Test build #142116 has finished for PR 32355 at commit f1e7ba0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-08-06T07:29:43Z

thanks, merging to master!

ulysses-you · 2021-08-06T08:40:03Z

thank you all for the reivew !

[SPARK-35221][SQL] Add join hint build side check

97029f2

github-actions bot added the SQL label Apr 27, 2021

cloud-fan reviewed Apr 27, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala Show resolved Hide resolved

maropu reviewed Apr 28, 2021

View reviewed changes

HyukjinKwon reviewed Aug 1, 2021

View reviewed changes

ulysses-you added 2 commits August 2, 2021 20:15

Merge branch 'master' of https://github.com/apache/spark into SPARK-3…

86e4059

…5221

test

f231e31

cloud-fan reviewed Aug 2, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala Show resolved Hide resolved

address comment

11c5700

cloud-fan reviewed Aug 3, 2021

View reviewed changes

name

307c8ac

cloud-fan reviewed Aug 3, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala Show resolved Hide resolved

cloud-fan reviewed Aug 3, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala Show resolved Hide resolved

cloud-fan approved these changes Aug 3, 2021

View reviewed changes

equi-join

2bb4200

ulysses-you changed the title ~~[SPARK-35221][SQL] Add join hint build side check~~ [SPARK-35221][SQL] Add join hint is supported check Aug 4, 2021

HyukjinKwon changed the title ~~[SPARK-35221][SQL] Add join hint is supported check~~ [SPARK-35221][SQL] Add the check of supported join hints Aug 4, 2021

cloud-fan reviewed Aug 5, 2021

View reviewed changes

ulysses-you added 2 commits August 5, 2021 22:05

address comment

d90dd66

size

5ac7952

cloud-fan reviewed Aug 5, 2021

View reviewed changes

size

f1e7ba0

cloud-fan closed this in c97fb68 Aug 6, 2021

ulysses-you deleted the SPARK-35221 branch August 6, 2021 08:39

ulysses-you mentioned this pull request Sep 2, 2021

[SPARK-36652][SQL] AQE dynamic join selection should not apply to non-equi join #33899

Closed

ulysses-you mentioned this pull request Jan 16, 2023

[SPARK-42064][SQL] Implement bloom filter join hint #39571

Closed

[SPARK-35221][SQL] Add the check of supported join hints #32355

[SPARK-35221][SQL] Add the check of supported join hints #32355

Conversation

ulysses-you commented Apr 27, 2021 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Apr 27, 2021

SparkQA commented Apr 27, 2021

ulysses-you commented Apr 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Aug 2, 2021

SparkQA commented Aug 2, 2021

SparkQA commented Aug 2, 2021

ulysses-you commented Aug 3, 2021

SparkQA commented Aug 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Aug 3, 2021

SparkQA commented Aug 3, 2021

SparkQA commented Aug 3, 2021

Choose a reason for hiding this comment

SparkQA commented Aug 3, 2021

SparkQA commented Aug 3, 2021

SparkQA commented Aug 3, 2021

SparkQA commented Aug 3, 2021

ulysses-you commented Aug 4, 2021

SparkQA commented Aug 4, 2021

SparkQA commented Aug 4, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Aug 5, 2021

SparkQA commented Aug 5, 2021

SparkQA commented Aug 5, 2021

SparkQA commented Aug 5, 2021

Choose a reason for hiding this comment

SparkQA commented Aug 5, 2021

SparkQA commented Aug 5, 2021

SparkQA commented Aug 6, 2021

SparkQA commented Aug 6, 2021

cloud-fan commented Aug 6, 2021

ulysses-you commented Aug 6, 2021

ulysses-you commented Apr 27, 2021 •

edited