Skip to content
Permalink
Browse files

[SPARK-23199][SQL] improved Removes repetition from group expressions…

… in Aggregate

## What changes were proposed in this pull request?

Currently, all Aggregate operations will go into RemoveRepetitionFromGroupExpressions, but there is no group expression or there is no duplicate group expression in group expression, we not need copy for logic plan.

## How was this patch tested?

the existed test case.

Author: caoxuewen <cao.xuewen@zte.com.cn>

Closes #20375 from heary-cao/RepetitionGroupExpressions.

(cherry picked from commit 54dd7cf)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
  • Loading branch information...
heary-cao authored and gatorsmile committed Jan 29, 2018
1 parent de66aba commit 4059454f979874caa9745861a2bcc60cac0bbffd
@@ -1302,8 +1302,12 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
*/
object RemoveRepetitionFromGroupExpressions extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
case a @ Aggregate(grouping, _, _) =>
case a @ Aggregate(grouping, _, _) if grouping.size > 1 =>
val newGrouping = ExpressionSet(grouping).toSeq
a.copy(groupingExpressions = newGrouping)
if (newGrouping.size == grouping.size) {
a
} else {
a.copy(groupingExpressions = newGrouping)
}
}
}
@@ -67,10 +67,9 @@ class AggregateOptimizeSuite extends PlanTest {
}

test("remove repetition in grouping expression") {
val input = LocalRelation('a.int, 'b.int, 'c.int)
val query = input.groupBy('a + 1, 'b + 2, Literal(1) + 'A, Literal(2) + 'B)(sum('c))
val query = testRelation.groupBy('a + 1, 'b + 2, Literal(1) + 'A, Literal(2) + 'B)(sum('c))
val optimized = Optimize.execute(analyzer.execute(query))
val correctAnswer = input.groupBy('a + 1, 'b + 2)(sum('c)).analyze
val correctAnswer = testRelation.groupBy('a + 1, 'b + 2)(sum('c)).analyze

comparePlans(optimized, correctAnswer)
}

0 comments on commit 4059454

Please sign in to comment.
You can’t perform that action at this time.