[SPARK-12957][SQL] Initial support for constraint propagation in SparkSQL #10844

sameeragarwal · 2016-01-20T08:17:36Z

Based on the semantics of a query, we can derive a number of data constraints on output of each (logical or physical) operator. For instance, if a filter defines ‘a > 10, we know that the output data of this filter satisfies 2 constraints:

‘a > 10
isNotNull(‘a)

This PR proposes a possible way of keeping track of these constraints and propagating them in the logical plan, which can then help us build more advanced optimizations (such as pruning redundant filters, optimizing joins, among others). We define constraints as a set of (implicitly conjunctive) expressions. For e.g., if a filter operator has constraints = Set(‘a > 10, ‘b < 100), it’s implied that the outputs satisfy both individual constraints (i.e., ‘a > 10 AND ‘b < 100).

Design Document: https://docs.google.com/a/databricks.com/document/d/1WQRgDurUBV9Y6CWOBS75PQIqJwT-6WftVa18xzm7nCo/edit?usp=sharing

SparkQA · 2016-01-20T08:27:07Z

Test build #49770 has finished for PR 10844 at commit f7251dd.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-20T10:36:16Z

Test build #49775 has finished for PR 10844 at commit 04ff99a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2016-01-20T17:06:44Z

Could you add the id of the JIRA ticket to the titlle?

Could you also add a description, explaining why we want this? Seems cool though!

sameeragarwal · 2016-01-22T02:52:04Z

@hvanhovell added, thanks!

gatorsmile · 2016-01-22T03:41:28Z

@sameeragarwal I just saw it. Thank you! I will do the code changes after this is merged.

SparkQA · 2016-01-26T09:38:33Z

Test build #50083 has finished for PR 10844 at commit 7fb2f9c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-27T03:49:59Z

Test build #50161 has finished for PR 10844 at commit f15ef96.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2016-01-27T23:33:12Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala

  self: PlanType =>

  def output: Seq[Attribute]

  /**
+   * Extracts the output property from a given child.
+   */
+  def extractConstraintsFromChild(child: QueryPlan[PlanType]): Set[Expression] = {


protected?

Also I'm not sure I get the scala doc. Maybe getReleventContraints is a better name? It is taking the constraints and removing those that don't apply anymore because we removed columns right?

SparkQA · 2016-01-29T11:09:52Z

Test build #50371 has finished for PR 10844 at commit 53be837.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sameeragarwal · 2016-01-29T20:09:50Z

Thanks @marmbrus, all comments addressed!

SparkQA · 2016-01-29T20:38:51Z

Test build #50404 has finished for PR 10844 at commit 8c6bb70.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sameeragarwal · 2016-01-29T23:26:06Z

test this please

marmbrus · 2016-01-29T23:31:56Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala

+   * operator. For example, if the output of this operator is column `a`, an example `constraints`
+   * can be `Set(a > 10, a < 20)`.
+   */
+  lazy val constraints: Set[Expression] = validConstraints


I would consider making this getRelevantConstraints(validConstraints) so that each implementor of validContraints does't have to remember to do the filter / canonicalization. They can just focus on augmenting or passing through constraints from children based on the operators semantics.

Great idea, added.

SparkQA · 2016-01-30T00:49:45Z

Test build #50417 has finished for PR 10844 at commit 8c6bb70.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sameeragarwal · 2016-01-30T02:34:49Z

Thanks @marmbrus, all comments addressed!

SparkQA · 2016-01-30T04:16:55Z

Test build #50429 has finished for PR 10844 at commit 302444f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2016-02-01T20:50:23Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala

+    }
+  }
+
+  def selfJoinResolved: Boolean = left.outputSet.intersect(right.outputSet).isEmpty


I think this was a merging mistake as its duplicated with the method below.

oops, fixed!

sameeragarwal · 2016-02-02T00:49:41Z

@marmbrus comments addressed!

sameeragarwal · 2016-02-02T01:11:15Z

test this please

SparkQA · 2016-02-02T02:49:47Z

Test build #50518 has finished for PR 10844 at commit b52742a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2016-02-02T10:15:03Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala

@@ -301,10 +301,12 @@ abstract class LeafNode extends LogicalPlan {
 /**
 * A logical plan node with single child.
 */
-abstract class UnaryNode extends LogicalPlan {
+abstract class UnaryNode extends LogicalPlan with PredicateHelper {


Looks like we do not need PredicateHelper at here? Maybe it is better to just with PredicateHelper for Filter and Join?

Sounds good, added this only for filter and join that needed splitConjunctivePredicates

sameeragarwal · 2016-02-02T22:38:41Z

comments addressed!

SparkQA · 2016-02-03T00:14:41Z

Test build #50603 has finished for PR 10844 at commit 2bd2735.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class QueryPlan[PlanType <: TreeNode[PlanType]] extends TreeNode[PlanType]
- abstract class UnaryNode extends LogicalPlan
- case class Filter(condition: Expression, child: LogicalPlan)

marmbrus · 2016-02-03T06:21:52Z

Thanks! Merging to master.

sameeragarwal added 4 commits January 20, 2016 00:09

initial framework

8d050df

Initial set of constraints

fed48b8

Constraint propagation in Set and Binary operators

76d2727

modify test

67e138d

fix tests

04ff99a

sameeragarwal force-pushed the constraints branch from f7251dd to 04ff99a Compare January 20, 2016 08:53

sameeragarwal changed the title ~~[WIP][SQL] Initial support for constraint propagation in SparkSQL~~ [WIP][SPARK-12957][SQL] Initial support for constraint propagation in SparkSQL Jan 22, 2016

sameeragarwal added 4 commits January 25, 2016 11:35

outstanding changes

7bde51d

upstream changes

494b28f

support union with multiple children

0c4c78b

join propagation

7fb2f9c

support all joins

f15ef96

marmbrus reviewed Jan 27, 2016
View reviewed changes

Michael's comments

53be837

sameeragarwal changed the title ~~[WIP][SPARK-12957][SQL] Initial support for constraint propagation in SparkSQL~~ [SPARK-12957][SQL] Initial support for constraint propagation in SparkSQL Jan 29, 2016

upstream changes

8c6bb70

marmbrus reviewed Jan 29, 2016
View reviewed changes

Michael's comments

302444f

marmbrus reviewed Feb 1, 2016
View reviewed changes

move constructIsNotNullConstraints in QueryPlan

b52742a

yhuai reviewed Feb 2, 2016
View reviewed changes

Yin's comments

2bd2735

sameeragarwal force-pushed the constraints branch from c1a0128 to 2bd2735 Compare February 2, 2016 22:38

asfgit closed this in 138c300 Feb 3, 2016

JoshRosen mentioned this pull request Jun 1, 2019

[SPARK-27915][SQL][WIP] Update logical Filter's output nullability based on IsNotNull conditions #24765

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-12957][SQL] Initial support for constraint propagation in SparkSQL #10844

[SPARK-12957][SQL] Initial support for constraint propagation in SparkSQL #10844

sameeragarwal commented Jan 20, 2016

SparkQA commented Jan 20, 2016

SparkQA commented Jan 20, 2016

hvanhovell commented Jan 20, 2016

sameeragarwal commented Jan 22, 2016

gatorsmile commented Jan 22, 2016

SparkQA commented Jan 26, 2016

SparkQA commented Jan 27, 2016

marmbrus Jan 27, 2016

SparkQA commented Jan 29, 2016

sameeragarwal commented Jan 29, 2016

SparkQA commented Jan 29, 2016

sameeragarwal commented Jan 29, 2016

marmbrus Jan 29, 2016

sameeragarwal Jan 30, 2016

SparkQA commented Jan 30, 2016

sameeragarwal commented Jan 30, 2016

SparkQA commented Jan 30, 2016

marmbrus Feb 1, 2016

sameeragarwal Feb 1, 2016

sameeragarwal commented Feb 2, 2016

sameeragarwal commented Feb 2, 2016

SparkQA commented Feb 2, 2016

yhuai Feb 2, 2016

sameeragarwal Feb 2, 2016

sameeragarwal commented Feb 2, 2016

SparkQA commented Feb 3, 2016

marmbrus commented Feb 3, 2016

[SPARK-12957][SQL] Initial support for constraint propagation in SparkSQL #10844

[SPARK-12957][SQL] Initial support for constraint propagation in SparkSQL #10844

Conversation

sameeragarwal commented Jan 20, 2016

SparkQA commented Jan 20, 2016

SparkQA commented Jan 20, 2016

hvanhovell commented Jan 20, 2016

sameeragarwal commented Jan 22, 2016

gatorsmile commented Jan 22, 2016

SparkQA commented Jan 26, 2016

SparkQA commented Jan 27, 2016

marmbrus Jan 27, 2016

Choose a reason for hiding this comment

SparkQA commented Jan 29, 2016

sameeragarwal commented Jan 29, 2016

SparkQA commented Jan 29, 2016

sameeragarwal commented Jan 29, 2016

marmbrus Jan 29, 2016

Choose a reason for hiding this comment

sameeragarwal Jan 30, 2016

Choose a reason for hiding this comment

SparkQA commented Jan 30, 2016

sameeragarwal commented Jan 30, 2016

SparkQA commented Jan 30, 2016

marmbrus Feb 1, 2016

Choose a reason for hiding this comment

sameeragarwal Feb 1, 2016

Choose a reason for hiding this comment

sameeragarwal commented Feb 2, 2016

sameeragarwal commented Feb 2, 2016

SparkQA commented Feb 2, 2016

yhuai Feb 2, 2016

Choose a reason for hiding this comment

sameeragarwal Feb 2, 2016

Choose a reason for hiding this comment

sameeragarwal commented Feb 2, 2016

SparkQA commented Feb 3, 2016

marmbrus commented Feb 3, 2016