[SPARK-11179] [SQL] Push filters through aggregate

[nitin2goyal]

Push conjunctive predicates though Aggregate operators when their references are a subset of the groupingExpressions.

Query plan before optimisation :-
Filter ((c#138L = 2) && (a#0 = 3))
Aggregate [a#0], [a#0,count(b#1) AS c#138L]
Project [a#0,b#1]
LocalRelation [a#0,b#1,c#2]

Query plan after optimisation :-
Filter (c#138L = 2)
Aggregate [a#0], [a#0,count(b#1) AS c#138L]
Filter (a#0 = 3)
Project [a#0,b#1]
LocalRelation [a#0,b#1,c#2]

[marmbrus]

Tests please. Look here for examples.

[nitin2goyal]

Added tests

[hvanhovell]

We could do a similar thing for window functions.

[rxin]

rather than checking for complete overlap, can we pull out the expressions for group by columns and push those down?

[nitin2goyal]

Good point. Incorporated.

[marmbrus]

ok to test

[marmbrus]

nit: indentation. I'm not sure we have a strict rule, but no indent is kinda hard to follow. I'd probably try to make it a tree if it fits?

case filter @ Filter(condition,
     aggregate @ Aggregate(groupingExpressions, aggregateExpressions, grandChild)) =>

or just 4 space indent?

[marmbrus]

This looks great, thanks for doing it!

Can you cleanup the title [SPARK-11179] [SQL] Push filters through aggregate
and the description: Push conjunctive predicates though Aggregate operators when their references are a subset of the groupingExpressions.

If you are feeling ambitious I'd include query plans before and after the optimization too.

These become the commit message when we use our merge tool.

[SparkQA]

Test build #43995 has finished for PR 9167 at commit 671fbb3.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[nitin2goyal]

Thanks for reviewing it Michael. Addressed 1 code review comment, fixed couple of scalastyle issues and cleaned up title and description in latest commit and this PR.

[SparkQA]

Test build #44002 has finished for PR 9167 at commit f422aa8.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[nitin2goyal]

@marmbrus Test failed again due to whitespace issue. I have already removed whitespace from that line and I am not getting scalastyle issue when compiling locally (don't see any whitespace in code review also). Am I missing something?

[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala:664:0: Whitespace at end of line

[marmbrus]

try sbt test:scalastyle

[rxin]

You can run local style tests by

dev/lint-scala

[nitin2goyal]

Somehow, I wasn't getting scalastyle issue on my branch with above 2 commands. Cloned apache spark master, cherry-picked my changes and then I got the error. Checked-in the whitespace removal. Please test this.

[SparkQA]

Test build #44041 has finished for PR 9167 at commit 82fc386.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Thanks, merging to master!