[SPARK-33472][SQL] Adjust RemoveRedundantSorts rule order #30373

allisonwang-db · 2020-11-13T23:58:18Z

What changes were proposed in this pull request?

This PR switched the order for the rule RemoveRedundantSorts and EnsureRequirements so that EnsureRequirements will be invoked before RemoveRedundantSorts to avoid IllegalArgumentException when instantiating PartitioningCollection.

Why are the changes needed?

RemoveRedundantSorts rule uses SparkPlan's outputPartitioning to check whether a sort node is redundant. Currently, it is added before EnsureRequirements. Since PartitioningCollection requires left and right partitioning to have the same number of partitions, which is not necessarily true before applying EnsureRequirements, the rule can fail with the following exception:

IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test

SparkQA · 2020-11-14T00:44:40Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35688/

SparkQA · 2020-11-14T01:07:47Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35688/

SparkQA · 2020-11-14T04:23:05Z

Test build #131085 has finished for PR 30373 at commit e6b0bb3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-11-14T13:43:57Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

      EnsureRequirements,
+      RemoveRedundantSorts,


Could you leave some comments here about why we need to put this rule after EnsureRequirements?

+1 for @maropu 's comment.

maropu · 2020-11-14T13:45:29Z

sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantSortsSuite.scala

@@ -135,6 +135,26 @@ abstract class RemoveRedundantSortsSuiteBase
      }
    }
  }
+
+  test("shuffled join with different left and right side partition numbers") {


nit: could you add the prefix: SPARK-33183:

maropu · 2020-11-14T13:47:47Z

sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantSortsSuite.scala

+      (0 to 100).toDF("key").createOrReplaceTempView("t2")
+
+      // left side partitioning: RangePartitioning(key ASC, 2)
+      // right side partitioning: UnknownPartitioning(0)


Could you add assert to check if the query below has the output partitions above?

dongjoon-hyun

@allisonwang-db . Please split this PR.

Please create a new JIRA for RemoveRedundantSorts. This looks worth.
Version update PR can be a follow-up.

SparkQA · 2020-11-18T02:34:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35844/

SparkQA · 2020-11-18T02:59:07Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35844/

dongjoon-hyun · 2020-11-18T03:50:58Z

cc @cloud-fan

SparkQA · 2020-11-18T04:48:11Z

Test build #131240 has finished for PR 30373 at commit fa6050a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-11-18T05:55:34Z

retest this please

SparkQA · 2020-11-18T07:24:09Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35859/

SparkQA · 2020-11-18T07:48:40Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35859/

SparkQA · 2020-11-18T08:05:01Z

Test build #131256 has finished for PR 30373 at commit fa6050a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-11-18T13:33:47Z

retest this please

SparkQA · 2020-11-18T14:19:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35887/

SparkQA · 2020-11-18T14:49:00Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35887/

SparkQA · 2020-11-18T16:41:16Z

Test build #131282 has finished for PR 30373 at commit fa6050a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-19T05:39:31Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35923/

SparkQA · 2020-11-19T06:11:59Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35923/

SparkQA · 2020-11-19T08:05:02Z

Test build #131319 has finished for PR 30373 at commit 4e684df.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-11-19T13:30:02Z

thanks, merging to master!

cloud-fan · 2020-11-19T13:30:26Z

@allisonwang-db can you create backport PRs for 2.4 and 3.0?

This PR switched the order for the rule `RemoveRedundantSorts` and `EnsureRequirements` so that `EnsureRequirements` will be invoked before `RemoveRedundantSorts` to avoid IllegalArgumentException when instantiating PartitioningCollection. `RemoveRedundantSorts` rule uses SparkPlan's `outputPartitioning` to check whether a sort node is redundant. Currently, it is added before `EnsureRequirements`. Since `PartitioningCollection` requires left and right partitioning to have the same number of partitions, which is not necessarily true before applying `EnsureRequirements`, the rule can fail with the following exception: ``` IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. ``` No Unit test Closes apache#30373 from allisonwang-db/sort-follow-up. Authored-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit a03c540) Signed-off-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com>

dongjoon-hyun · 2020-11-20T17:26:40Z

Thank you, @allisonwang-db , @maropu , @cloud-fan !

Backport #30373 for branch-3.0. ### What changes were proposed in this pull request? This PR switched the order for the rule `RemoveRedundantSorts` and `EnsureRequirements` so that `EnsureRequirements` will be invoked before `RemoveRedundantSorts` to avoid IllegalArgumentException when instantiating PartitioningCollection. ### Why are the changes needed? `RemoveRedundantSorts` rule uses SparkPlan's `outputPartitioning` to check whether a sort node is redundant. Currently, it is added before `EnsureRequirements`. Since `PartitioningCollection` requires left and right partitioning to have the same number of partitions, which is not necessarily true before applying `EnsureRequirements`, the rule can fail with the following exception: ``` IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit test Closes #30438 from allisonwang-db/spark-33472-3.0. Authored-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Backport #30373 for branch-2.4. ### What changes were proposed in this pull request? This PR switched the order for the rule `RemoveRedundantSorts` and `EnsureRequirements` so that `EnsureRequirements` will be invoked before `RemoveRedundantSorts` to avoid IllegalArgumentException when instantiating PartitioningCollection. ### Why are the changes needed? `RemoveRedundantSorts` rule uses SparkPlan's `outputPartitioning` to check whether a sort node is redundant. Currently, it is added before `EnsureRequirements`. Since `PartitioningCollection` requires left and right partitioning to have the same number of partitions, which is not necessarily true before applying `EnsureRequirements`, the rule can fail with the following exception: ``` IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit test Closes #30437 from allisonwang-db/spark-33472-2.4. Authored-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

gatorsmile · 2020-11-21T17:30:17Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

      EnsureRequirements,
+      // `RemoveRedundantSorts` needs to be added before `EnsureRequirements` to guarantee the same


before -> after ?

Ah, I missed it. @allisonwang-db could you fix it?

Thanks for catching it! Will create a fix.

### What changes were proposed in this pull request? This PR is a follow-up for #30373 that updates the comment for RemoveRedundantSorts in QueryExecution. ### Why are the changes needed? To update an incorrect comment. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N/A Closes #30584 from allisonwang-db/spark-33472-followup. Authored-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR is a follow-up for #30373 that updates the comment for RemoveRedundantSorts in QueryExecution. ### Why are the changes needed? To update an incorrect comment. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N/A Closes #30584 from allisonwang-db/spark-33472-followup. Authored-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 960d6af) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actions bot added the SQL label Nov 13, 2020

maropu reviewed Nov 14, 2020

View reviewed changes

dongjoon-hyun requested changes Nov 17, 2020

View reviewed changes

allisonwang-db changed the title ~~[SPARK-33183][SQL][FOLLOW-UP] Adjust RemoveRedundantSorts rule order and update config version~~ [SPARK-33472][SQL] Adjust RemoveRedundantSorts rule order Nov 18, 2020

allisonwang-db added 3 commits November 18, 2020 20:37

update

117dc6d

address comments

9d28f40

fix test

4e684df

allisonwang-db force-pushed the sort-follow-up branch from fa6050a to 4e684df Compare November 19, 2020 04:50

cloud-fan closed this in a03c540 Nov 19, 2020

This was referenced Nov 20, 2020

[SPARK-33472][SQL][2.4] Adjust RemoveRedundantSorts rule order #30437

Closed

[SPARK-33472][SQL][3.0] Adjust RemoveRedundantSorts rule order #30438

Closed

gatorsmile reviewed Nov 21, 2020

View reviewed changes

allisonwang-db mentioned this pull request Dec 3, 2020

[SPARK-33472][SQL][FOLLOW-UP] Update RemoveRedundantSorts comment #30584

Closed

allisonwang-db deleted the sort-follow-up branch January 19, 2024 01:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-33472][SQL] Adjust RemoveRedundantSorts rule order #30373

[SPARK-33472][SQL] Adjust RemoveRedundantSorts rule order #30373

allisonwang-db commented Nov 13, 2020 •

edited

Loading

SparkQA commented Nov 14, 2020

SparkQA commented Nov 14, 2020

SparkQA commented Nov 14, 2020

maropu Nov 14, 2020

dongjoon-hyun Nov 17, 2020

maropu Nov 14, 2020

maropu Nov 14, 2020

dongjoon-hyun left a comment •

edited

Loading

SparkQA commented Nov 18, 2020

SparkQA commented Nov 18, 2020

dongjoon-hyun commented Nov 18, 2020

SparkQA commented Nov 18, 2020

cloud-fan commented Nov 18, 2020

SparkQA commented Nov 18, 2020

SparkQA commented Nov 18, 2020

SparkQA commented Nov 18, 2020

cloud-fan commented Nov 18, 2020

SparkQA commented Nov 18, 2020

SparkQA commented Nov 18, 2020

SparkQA commented Nov 18, 2020

SparkQA commented Nov 19, 2020

SparkQA commented Nov 19, 2020

SparkQA commented Nov 19, 2020

cloud-fan commented Nov 19, 2020

cloud-fan commented Nov 19, 2020

dongjoon-hyun commented Nov 20, 2020

gatorsmile Nov 21, 2020

maropu Nov 22, 2020

allisonwang-db Nov 24, 2020

		EnsureRequirements,
		// `RemoveRedundantSorts` needs to be added before `EnsureRequirements` to guarantee the same

[SPARK-33472][SQL] Adjust RemoveRedundantSorts rule order #30373

[SPARK-33472][SQL] Adjust RemoveRedundantSorts rule order #30373

Conversation

allisonwang-db commented Nov 13, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Nov 14, 2020

SparkQA commented Nov 14, 2020

SparkQA commented Nov 14, 2020

maropu Nov 14, 2020

Choose a reason for hiding this comment

dongjoon-hyun Nov 17, 2020

Choose a reason for hiding this comment

maropu Nov 14, 2020

Choose a reason for hiding this comment

maropu Nov 14, 2020

Choose a reason for hiding this comment

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

SparkQA commented Nov 18, 2020

SparkQA commented Nov 18, 2020

dongjoon-hyun commented Nov 18, 2020

SparkQA commented Nov 18, 2020

cloud-fan commented Nov 18, 2020

SparkQA commented Nov 18, 2020

SparkQA commented Nov 18, 2020

SparkQA commented Nov 18, 2020

cloud-fan commented Nov 18, 2020

SparkQA commented Nov 18, 2020

SparkQA commented Nov 18, 2020

SparkQA commented Nov 18, 2020

SparkQA commented Nov 19, 2020

SparkQA commented Nov 19, 2020

SparkQA commented Nov 19, 2020

cloud-fan commented Nov 19, 2020

cloud-fan commented Nov 19, 2020

dongjoon-hyun commented Nov 20, 2020

gatorsmile Nov 21, 2020

Choose a reason for hiding this comment

maropu Nov 22, 2020

Choose a reason for hiding this comment

allisonwang-db Nov 24, 2020

Choose a reason for hiding this comment

allisonwang-db commented Nov 13, 2020 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading