Skip to content

Commit

Permalink
Don't propagate empty relation through aggregate for streaming.
Browse files Browse the repository at this point in the history
  • Loading branch information
jose-torres committed Aug 25, 2017
1 parent d815e56 commit a25534e
Showing 1 changed file with 3 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,12 @@ object PropagateEmptyRelation extends Rule[LogicalPlan] with PredicateHelper {
case _: RepartitionByExpression => empty(p)
// An aggregate with non-empty group expression will return one output row per group when the
// input to the aggregate is not empty. If the input to the aggregate is empty then all groups
// will be empty and thus the output will be empty.
// will be empty and thus the output will be empty. If we're working on batch data, we can
// then treat the aggregate as redundant.
//
// If the grouping expressions are empty, however, then the aggregate will always produce a
// single output row and thus we cannot propagate the EmptyRelation.
case Aggregate(ge, _, _) if ge.nonEmpty => empty(p)
case Aggregate(ge, _, _) if ge.nonEmpty and !p.isStreaming => empty(p)
// Generators like Hive-style UDTF may return their records within `close`.
case Generate(_: Explode, _, _, _, _, _) => empty(p)
case _ => p
Expand Down

0 comments on commit a25534e

Please sign in to comment.