Skip to content

Commit

Permalink
[SPARK-22141][BACKPORT][SQL] Propagate empty relation before checking…
Browse files Browse the repository at this point in the history
… Cartesian products

Back port #19362 to branch-2.2

## What changes were proposed in this pull request?

When inferring constraints from children, Join's condition can be simplified as None.
For example,
```
val testRelation = LocalRelation('a.int)
val x = testRelation.as("x")
val y = testRelation.where($"a" === 2 && !($"a" === 2)).as("y")
x.join.where($"x.a" === $"y.a")
```
The plan will become
```
Join Inner
:- LocalRelation <empty>, [a#23]
+- LocalRelation <empty>, [a#224]
```
And the Cartesian products check will throw exception for above plan.

Propagate empty relation before checking Cartesian products, and the issue is resolved.

## How was this patch tested?

Unit test

Author: Wang Gengliang <ltnwgl@gmail.com>

Closes #19366 from gengliangwang/branch-2.2.
  • Loading branch information
gengliangwang authored and hvanhovell committed Sep 27, 2017
1 parent b0f30b5 commit a406473
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -113,8 +113,6 @@ abstract class Optimizer(sessionCatalog: SessionCatalog, conf: SQLConf)
SimplifyCreateArrayOps,
SimplifyCreateMapOps) ++
extendedOperatorOptimizationRules: _*) ::
Batch("Check Cartesian Products", Once,
CheckCartesianProducts(conf)) ::
Batch("Join Reorder", Once,
CostBasedJoinReorder(conf)) ::
Batch("Decimal Optimizations", fixedPoint,
Expand All @@ -125,6 +123,8 @@ abstract class Optimizer(sessionCatalog: SessionCatalog, conf: SQLConf)
Batch("LocalRelation", fixedPoint,
ConvertToLocalRelation,
PropagateEmptyRelation) ::
Batch("Check Cartesian Products", Once,
CheckCartesianProducts(conf)) ::
Batch("OptimizeCodegen", Once,
OptimizeCodegen(conf)) ::
Batch("RewriteSubquery", Once,
Expand Down
8 changes: 8 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,14 @@ class JoinSuite extends QueryTest with SharedSQLContext {
Nil)
}

test("SPARK-22141: Propagate empty relation before checking Cartesian products") {
Seq("inner", "left", "right", "left_outer", "right_outer", "full_outer").foreach { joinType =>
val x = testData2.where($"a" === 2 && !($"a" === 2)).as("x")
val y = testData2.where($"a" === 1 && !($"a" === 1)).as("y")
checkAnswer(x.join(y, Seq.empty, joinType), Nil)
}
}

test("big inner join, 4 matches per row") {
val bigData = testData.union(testData).union(testData).union(testData)
val bigDataX = bigData.as("x")
Expand Down

0 comments on commit a406473

Please sign in to comment.