Skip to content

[SPARK-42500][SQL] ConstantPropagation supports more cases #40093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[SPARK-42500][SQL] ConstantPropagation supports more cases #40093

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,14 +200,20 @@ object ConstantPropagation extends Rule[LogicalPlan] {

private def replaceConstants(condition: Expression, equalityPredicates: EqualityPredicates)
: Expression = {
val constantsMap = AttributeMap(equalityPredicates.map(_._1))
val predicates = equalityPredicates.map(_._2).toSet
def replaceConstants0(expression: Expression) = expression transform {
val allConstantsMap = AttributeMap(equalityPredicates.map(_._1))
val allPredicates = equalityPredicates.map(_._2).toSet
def replaceConstants0(
expression: Expression, constantsMap: AttributeMap[Literal]) = expression transform {
case a: AttributeReference => constantsMap.getOrElse(a, a)
}
condition transform {
case e @ EqualTo(_, _) if !predicates.contains(e) => replaceConstants0(e)
case e @ EqualNullSafe(_, _) if !predicates.contains(e) => replaceConstants0(e)
case b: BinaryComparison =>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To support other binary comparisons. For example: >, <.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we look at the commit history and understand why constant propagation only supports equality predicates before?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The initial commit did not support it. It seems other binary comparisons were not considered at the time:
#17993

Later we tried to enhance it. It seems it is not easy because there are too many changes:
#24553

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @peter-toth

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can replace constants in other expressions too, #24553 does the same (and more).

Copy link
Contributor

@peter-toth peter-toth Mar 1, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I'm not sure I get the issue. If a complex expression is equal to a constant why we can't use that constant where the complex expression appears? I don't think it matters if some parts of the complex expression can also be replaced to another constat. The replace happens using transformDown so larger complex expressions are replaced to constants first. E.g. a = 1 AND (a + b) = 1 AND (a + b) > 2 -> a = 1 AND (1 + b) = 1 AND 1 > 2 -> false makes sense to me.

  2. Yes we can. The reason why I didn't do those kind of shortcuts in ConstantPropagation in my PR is because there are other, similar cases where it isn't that easy to do a shortcut (or simplification has been handled in other rules so why doing it here as well) . E.g. a = 1 AND a > 2 -> a = 1 AND 1 > 2 -> false where the 2nd step (1 > 2 -> false) could also be handled in ConstantPropagation but those foldable expressions are already handled in ConstantFolding and then in BooleanSimplification when contant propagation has been done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me open a PR tomorrow.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've opened #40268.
I didn't want to overcomplicate the change so took only a few parts from #24553. But if you think 1. or 2. or any other feature from the original PR is also needed I can add it to the PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan This is why conflicting equality predicates are not supported: #17993 (comment)
It is correct if it only optimize on the filter condition.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sidenote: In my old PR (https://github.com/apache/spark/pull/24553/files#diff-d43484d56a4d9991066b5c00d12ec2465c75131e055fc02ee7fb6dfd45b5006fR76-R79) I proposed #27309 to detect equijoins better and to allow constant propagation in Joins too.

if (!allPredicates.contains(b)) {
replaceConstants0(b, allConstantsMap)
} else {
val excludedEqualityPredicates = equalityPredicates.filterNot(_._2.semanticEquals(b))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exclude current binary comparison to support the following case:

a = 1 and a = 2 ==> 2 = 1 and 1 = 2

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't look like constant propagation but a different optimization to simplify predicates:

  • a = 1 and a = 2 ==> false
  • a > 1 and a > 2 ==> a > 2

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR does not support a > 1 and a > 2 ==> a > 2. It must contain an equality predicate.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we exclude it first? We should add a new optimizer rule later to do more advanced predicate simplification.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean only support a = 1 and a = 2 ==> 2 = 1 and 1 = 2 ==> false, and do not support a = 1 and b > a + 2 ==> a = 1 && b > 3?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvm, the rule name is ConstantPropagation, so both optimizations fit it.

replaceConstants0(b, AttributeMap(excludedEqualityPredicates.map(_._1)))
}
}
}
}
Expand Down
33 changes: 31 additions & 2 deletions ...yst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
import org.apache.spark.sql.catalyst.dsl.expressions._
import org.apache.spark.sql.catalyst.dsl.plans._
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
import org.apache.spark.sql.catalyst.plans.PlanTest
import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan}
import org.apache.spark.sql.catalyst.rules.RuleExecutor
Expand Down Expand Up @@ -159,8 +160,9 @@ class ConstantPropagationSuite extends PlanTest {
columnA === Literal(1) && columnA === Literal(2) && columnB === Add(columnA, Literal(3)))

val correctAnswer = testRelation
.select(columnA)
.where(columnA === Literal(1) && columnA === Literal(2) && columnB === Literal(5)).analyze
.select(columnA, columnB)
.where(FalseLiteral)
.select(columnA).analyze

comparePlans(Optimize.execute(query.analyze), correctAnswer)
}
Expand All @@ -186,4 +188,31 @@ class ConstantPropagationSuite extends PlanTest {
.analyze
comparePlans(Optimize.execute(query2), correctAnswer2)
}

test("SPARK-42500: ConstantPropagation supports more cases") {
comparePlans(
Optimize.execute(testRelation.where(columnA === 1 && columnB > columnA + 2).analyze),
testRelation.where(columnA === 1 && columnB > 3).analyze)

comparePlans(
Optimize.execute(testRelation.where(columnA === 1 && columnA === 2).analyze),
testRelation.where(FalseLiteral).analyze)

comparePlans(
Optimize.execute(testRelation.where(columnA === 1 && columnA === columnA + 2).analyze),
testRelation.where(FalseLiteral).analyze)

comparePlans(
Optimize.execute(
testRelation.where((columnA === 1 || columnB === 2) && columnB === 1).analyze),
testRelation.where(columnA === 1 && columnB === 1).analyze)

comparePlans(
Optimize.execute(testRelation.where(columnA === 1 && columnA === 1).analyze),
testRelation.where(columnA === 1).analyze)

comparePlans(
Optimize.execute(testRelation.where(Not(columnA === 1 && columnA === columnA + 2)).analyze),
testRelation.where(Not(columnA === 1) || Not(columnA === columnA + 2)).analyze)
}
}