Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-7824][SQL]Collapsing operator reordering and constant folding into a single batch to push down the single side. #6351

Closed
wants to merge 27 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
c3f046f
Merge pull request #1 from apache/master
pzzs Mar 2, 2015
cb1852d
Merge pull request #2 from apache/master
pzzs Mar 5, 2015
c87e8b6
Merge pull request #3 from apache/master
pzzs Mar 10, 2015
161cae3
Merge pull request #4 from apache/master
pzzs Mar 12, 2015
98b134f
Merge pull request #5 from apache/master
pzzs Mar 19, 2015
d00303b
Merge pull request #6 from apache/master
pzzs Mar 24, 2015
802261c
Merge pull request #7 from apache/master
pzzs Mar 26, 2015
34b1a9a
Merge pull request #8 from apache/master
pzzs Apr 8, 2015
f61210c
Merge pull request #9 from apache/master
pzzs Apr 17, 2015
f12fa50
Merge pull request #10 from apache/master
pzzs Apr 21, 2015
f03fe7f
Merge pull request #12 from apache/master
pzzs Apr 29, 2015
14952e2
Merge pull request #13 from apache/master
pzzs May 7, 2015
ab8e9a6
Merge pull request #14 from apache/master
pzzs May 22, 2015
fa65718
Update Optimizer.scala
pzzs May 22, 2015
d98bc35
Update FilterPushdownSuite.scala
pzzs May 22, 2015
8df716a
Update FilterPushdownSuite.scala
pzzs May 23, 2015
b01e622
Merge pull request #15 from apache/master
pzzs May 26, 2015
be6b1d5
Update Optimizer.scala
pzzs May 27, 2015
f2ee5fe
Update Optimizer.scala
pzzs May 27, 2015
11beb61
Update FilterPushdownSuite.scala
pzzs May 27, 2015
a04ffae
Update Optimizer.scala
pzzs May 27, 2015
ae3af6d
Update FilterPushdownSuite.scala
pzzs Jun 1, 2015
c529d9f
Update FilterPushdownSuite.scala
pzzs Jun 1, 2015
f8b9314
Update FilterPushdownSuite.scala
pzzs Jun 1, 2015
0ba5f42
Update Optimizer.scala
pzzs Jun 10, 2015
7bc7d28
Merge pull request #17 from apache/master
pzzs Jun 10, 2015
20de7be
Update Optimizer.scala
pzzs Jun 10, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -38,21 +38,20 @@ object DefaultOptimizer extends Optimizer {
EliminateSubQueries) ::
Batch("Distinct", FixedPoint(100),
ReplaceDistinctWithAggregate) ::
Batch("Operator Reordering", FixedPoint(100),
Batch("Operator Optimizations", FixedPoint(100),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about "Operator Reordering and ConstantFolding"

UnionPushdown,
CombineFilters,
PushPredicateThroughProject,
PushPredicateThroughJoin,
PushPredicateThroughGenerate,
ColumnPruning,
ProjectCollapsing,
CombineLimits) ::
Batch("ConstantFolding", FixedPoint(100),
CombineLimits,
NullPropagation,
OptimizeIn,
ConstantFolding,
LikeSimplification,
BooleanSimplification,
PushPredicateThroughJoin,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't need to change the rule order inside a batch with fixed point. Rules within a single batch shouldn't be sensitive to execution order.

SimplifyFilters,
SimplifyCasts,
SimplifyCaseConversionExpressions) ::
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ class FilterPushdownSuite extends PlanTest {
Batch("Filter Pushdown", Once,
CombineFilters,
PushPredicateThroughProject,
BooleanSimplification,
PushPredicateThroughJoin,
PushPredicateThroughGenerate,
ColumnPruning,
Expand Down Expand Up @@ -156,11 +157,9 @@ class FilterPushdownSuite extends PlanTest {
.where('a === 1 && 'a === 2)
.select('a).analyze


comparePlans(optimized, correctAnswer)
}


test("joins: push to either side") {
val x = testRelation.subquery('x)
val y = testRelation.subquery('y)
Expand Down Expand Up @@ -198,6 +197,25 @@ class FilterPushdownSuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}

test("joins: push to one side after transformCondition") {
val x = testRelation.subquery('x)
val y = testRelation1.subquery('y)

val originalQuery = {
x.join(y)
.where(("x.a".attr === 1 && "y.d".attr === "x.b".attr) ||
("x.a".attr === 1 && "y.d".attr === "x.c".attr))
}

val optimized = Optimize.execute(originalQuery.analyze)
val left = testRelation.where('a === 1)
val right = testRelation1
val correctAnswer =
left.join(right, condition = Some("d".attr === "b".attr || "d".attr === "c".attr)).analyze

comparePlans(optimized, correctAnswer)
}

test("joins: rewrite filter to push to either side") {
val x = testRelation.subquery('x)
val y = testRelation.subquery('y)
Expand Down Expand Up @@ -563,17 +581,16 @@ class FilterPushdownSuite extends PlanTest {
// push down invalid
val originalQuery1 = {
x.select('a, 'b)
.sortBy(SortOrder('a, Ascending))
.select('b)
.sortBy(SortOrder('a, Ascending))
.select('b)
}

val optimized1 = Optimize.execute(originalQuery1.analyze)
val correctAnswer1 =
x.select('a, 'b)
.sortBy(SortOrder('a, Ascending))
.select('b).analyze
.sortBy(SortOrder('a, Ascending))
.select('b).analyze

comparePlans(optimized1, analysis.EliminateSubQueries(correctAnswer1))

}
}