[SPARK-24953] [SQL] Prune a branch in CaseWhen if previously seen
#21904
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #93705 has finished for PR 21904 at commit
|
mgaido91
reviewed
Jul 28, 2018
| @@ -416,6 +416,29 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper { | |||
| // these branches can be pruned away | |||
| val (h, t) = branches.span(_._1 != TrueLiteral) | |||
| CaseWhen( h :+ t.head, None) | |||
|
|
|||
| case e @ CaseWhen(branches, _) => | |||
| val newBranches = branches.foldLeft(List[(Expression, Expression)]()) { | |||
There was a problem hiding this comment.
what about using ArrayBuffer? So we don't have to recreate the List at every iteration...
| newBranches | ||
| } else if (newBranches.nonEmpty && newBranches.last._2.semanticEquals(branch._2)) { | ||
| // If the outputs of two adjacent branches are the same, two branches can be combined. | ||
| newBranches.take(newBranches.length - 1) |
viirya
reviewed
Jul 29, 2018
| } else if (newBranches.nonEmpty && newBranches.last._2.semanticEquals(branch._2)) { | ||
| // If the outputs of two adjacent branches are the same, two branches can be combined. | ||
| newBranches.take(newBranches.length - 1) | ||
| .:+((Or(newBranches.last._1, branch._1), newBranches.last._2)) |
There was a problem hiding this comment.
For example, the following case can be benefitted from this rule,
CaseWhen((UnresolvedAttribute("a"), Literal(1)) ::
(Not(UnresolvedAttribute("a")), Literal(1)) ::
(LessThan(Rand(1), Literal(0.5)), Literal(3)) ::
(NonFoldableLiteral(true), Literal(4)) ::
(NonFoldableLiteral(false), Literal(5)) ::
Nil,
None),
Literal(1))| // TODO: In fact, if a condition is a sub-condition of the previous one, | ||
| // TODO: it can be pruned. This is less strict and can be implemented | ||
| // TODO: by decomposing seen conditions. | ||
| newBranches |
There was a problem hiding this comment.
This seems good as the branch is useless. Removing it should simplify code and query plan.
viirya
reviewed
Jul 30, 2018
| e.copy(branches = pruneSeenBranches(branches).get) | ||
|
|
||
| case e @ CaseWhen(branches, _) if combineAdjacentBranches(branches).nonEmpty => | ||
| e.copy(branches = combineAdjacentBranches(branches).get) |
There was a problem hiding this comment.
Maybe:
case e @ CaseWhen(branches, _) =>
val prunedBranches = pruneSeenBranches(branches).getOrElse(branches)
val newBranches = combineAdjacentBranches(prunedBranches).getOrElse(prunedBranches)
if (newBranches.length < branches.length) {
e.copy(branches = newBranches)
} else {
e
}There was a problem hiding this comment.
If pruneSeenBranches and combineAdjacentBranches both return input branches if no changes applied, this maybe more simplified:
case e @ CaseWhen(branches, _) =>
val prunedBranches = pruneSeenBranches(branches)
val newBranches = combineAdjacentBranches(prunedBranches)
if (newBranches.length < branches.length) {
e.copy(branches = newBranches)
} else {
e
}There was a problem hiding this comment.
This will work. But my only concern is that this will be caught all the CaseWhen cases; as a result, no more rule can be added after this.
|
Test build #93804 has finished for PR 21904 at commit
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
If a condition in a branch is previously seen, this branch can be pruned.
If the outputs of two adjacent branches are the same, two branches can be combined.
How was this patch tested?
Tests added.