[SPARK-35544][SQL] Add tree pattern pruning to Analyzer rules #32686

sigmod · 2021-05-27T18:52:18Z

What changes were proposed in this pull request?

Added the following TreePattern enums:

AGGREGATE_EXPRESSION
ALIAS
GROUPING_ANALYTICS
GENERATOR
HIGH_ORDER_FUNCTION
LAMBDA_FUNCTION
NEW_INSTANCE
PIVOT
PYTHON_UDF
TIME_WINDOW
TIME_ZONE_AWARE_EXPRESSION
UP_CAST
COMMAND
EVENT_TIME_WATERMARK
UNRESOLVED_RELATION
WITH_WINDOW_DEFINITION
UNRESOLVED_ALIAS
UNRESOLVED_ATTRIBUTE
UNRESOLVED_DESERIALIZER
UNRESOLVED_ORDINAL
UNRESOLVED_FUNCTION
UNRESOLVED_HINT
UNRESOLVED_SUBQUERY_COLUMN_ALIAS
UNRESOLVED_FUNC

Added tree pattern pruning to the following Analyzer rules:

ResolveBinaryArithmetic
WindowsSubstitution
ResolveAliases
ResolveGroupingAnalytics
ResolvePivot
ResolveOrdinalInOrderByAndGroupBy
LookupFunction
ResolveSubquery
ResolveSubqueryColumnAliases
ApplyCharTypePadding
UpdateOuterReferences
ResolveCreateNamedStruct
TimeWindowing
CleanupAliases
EliminateUnions
EliminateSubqueryAliases
HandleAnalysisOnlyCommand
ResolveNewInstances
ResolveUpCast
ResolveDeserializer
ResolveOutputRelation
ResolveEncodersInUDF
HandleNullInputsForUDF
ResolveGenerate
ExtractGenerator
GlobalAggregates
ResolveAggregateFunctions

Why are the changes needed?

Reduce the number of tree traversals and hence improve the query compilation latency.

How was this patch tested?

Existing tests.
Performance diff:
<style type="text/css"></style>

	Baseline	Experiment	Experiment/Baseline
ResolveBinaryArithmetic	43264874	34707117	0.80
WindowsSubstitution	3322996	2734192	0.82
ResolveAliases	24859263	21359941	0.86
ResolveGroupingAnalytics	39249143	25417569	0.80
ResolvePivot	6393408	2843314	0.44
ResolveOrdinalInOrderByAndGroupBy	10750806	3386715	0.32
LookupFunction	22087384	15481294	0.70
ResolveSubquery	1129139340	944402323	0.84
ResolveSubqueryColumnAliases	5055038	2808210	0.56
ApplyCharTypePadding	76285576	63785681	0.84
UpdateOuterReferences	6548321	3092539	0.47
ResolveCreateNamedStruct	38111477	17350249	0.46
TimeWindowing	41694190	3739134	0.09
CleanupAliases	48683506	39584921	0.81
EliminateUnions	3405069	2372506	0.70
EliminateSubqueryAliases	9626649	9518216	0.99
HandleAnalysisOnlyCommand	2562123	2661432	1.04
ResolveNewInstances	16208966	1982314	0.12
ResolveUpCast	14067843	1868615	0.13
ResolveDeserializer	17991103	2320308	0.13
ResolveOutputRelation	5815277	2088787	0.36
ResolveEncodersInUDF	14182892	1045113	0.07
HandleNullInputsForUDF	19850838	`1329528`	0.07
ResolveGenerate	5587345	1953192	0.35
ExtractGenerator	120378046	3386286	0.03
GlobalAggregates	16510455	13553155	0.82
ResolveAggregateFunctions	1041848509	828049280	0.79

SparkQA · 2021-05-27T19:36:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43546/

SparkQA · 2021-05-27T20:09:23Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43546/

SparkQA · 2021-05-27T20:09:53Z

Test build #139029 has finished for PR 32686 at commit b7d966e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-27T20:24:31Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43547/

SparkQA · 2021-05-27T20:49:26Z

Test build #139031 has finished for PR 32686 at commit c001574.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-27T20:50:06Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43549/

SparkQA · 2021-05-27T21:54:09Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43547/

SparkQA · 2021-05-28T12:58:05Z

Test build #139054 has finished for PR 32686 at commit f57f0fb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-28T20:37:00Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43591/

SparkQA · 2021-05-28T21:11:01Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43591/

SparkQA · 2021-05-29T00:10:20Z

Test build #139070 has finished for PR 32686 at commit ad9dbed.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-31T05:49:34Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43617/

SparkQA · 2021-05-31T06:26:29Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43617/

sigmod · 2021-05-31T08:45:23Z

@dbaliafroozeh @gengliangwang @hvanhovell @maryannxue this PR is ready for review. Thanks!

SparkQA · 2021-05-31T08:59:21Z

Test build #139096 has finished for PR 32686 at commit 0542922.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-31T09:45:58Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43624/

SparkQA · 2021-05-31T10:19:04Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43624/

SparkQA · 2021-05-31T14:01:41Z

Test build #139104 has finished for PR 32686 at commit 53d07ee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell

LGTM

hvanhovell · 2021-05-31T14:41:32Z

One small ergonomic comment. I would be great if we can create some shorthand for the function closures. I would probably make the in individual value be matcher for itself (if Enumeration allows subclassing of the Value class), and create a bunch of functions that allow you to compose them, e.g.: any, all, ...

gengliangwang · 2021-05-31T13:33:24Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

@@ -423,7 +424,9 @@ class Analyzer(override val catalogManager: CatalogManager)
   */
  object ResolveAliases extends Rule[LogicalPlan] {
    private def assignAliases(exprs: Seq[NamedExpression]) = {
-      exprs.map(_.transformUp { case u @ UnresolvedAlias(child, optGenAliasFunc) =>
+      exprs.map(_.transformUpWithPruning(_.containsPattern(UNRESOLVED_ALIAS))
+        {


Nit: the indention looks wired here. Shall we move the { in the above line?

gengliangwang · 2021-05-31T13:51:20Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

@@ -1876,7 +1879,7 @@ class Analyzer(override val catalogManager: CatalogManager)
    private def allowGroupByAlias: Boolean = conf.groupByAliases && !conf.ansiEnabled

    override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUpWithPruning(
-      AlwaysProcess.fn, ruleId) {
+      _.containsAllPatterns(AGGREGATE, UNRESOLVED_ATTRIBUTE), ruleId) {


Let's add comment saying that mayResolveAttrByAggregateExprs requires the TreePattern UNRESOLVED_ATTRIBUTE

gengliangwang · 2021-05-31T14:07:28Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

@@ -3736,7 +3744,8 @@ object EliminateUnions extends Rule[LogicalPlan] {
 * rule can't work for those parameters.
 */
 object CleanupAliases extends Rule[LogicalPlan] with AliasHelper {
-  override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUpWithPruning(
+    _.containsPattern(ALIAS)) {


It seems that we need to set the Tree Pattern of MultiAlias.

Done. Thanks for the catch!

gengliangwang · 2021-05-31T14:21:23Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala

@@ -117,6 +120,7 @@ case class AggregateExpression(
    UnresolvedAttribute(aggregateFunction.toString)
  }

+


Unnecessary change.

sigmod · 2021-06-01T00:33:22Z

One small ergonomic comment. I would be great if we can create some shorthand for the function closures. I would probably make the in individual value be matcher for itself (if Enumeration allows subclassing of the Value class), and create a bunch of functions that allow you to compose them, e.g.: any, all, ...

I'm not sure what the transformWithPruning interface exactly looks like. IIUC, transformWithPruning may still not be able to just take a composed pattern instead of a lambda, because we also have and, or, not over all, any -- even though they're not frequent. If we'd like to put and, or, not into patterns, it sounds a bit complex, as we need to be able to process a tree of such compositions.

Anyway, thanks for the suggestion. I'll think about whether there's a simpler approach and may address it subsequent PRs.

SparkQA · 2021-06-01T01:31:41Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43645/

gengliangwang · 2021-06-01T03:39:13Z

Thanks, merging to master

SparkQA · 2021-06-01T04:26:42Z

Test build #139125 has finished for PR 32686 at commit 8252a6a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sigmod added 9 commits May 23, 2021 16:54

update

562c260

Merge branch 'master' into all_expressions

773bb96

Merge branch 'master' into all_expressions

e1a2ccb

update

3f2dde6

update

0a3e93c

update

330a4d0

update

3a29005

update

39f5811

update

66e4f34

github-actions bot added SQL STRUCTURED STREAMING labels May 27, 2021

update

b7d966e

sigmod changed the title ~~[WIP][SPARK-35544] Add tree pattern pruning to Analyzer rules~~ [WIP][SPARK-35544][SQL] Add tree pattern pruning to Analyzer rules May 27, 2021

update

c001574

sigmod added 3 commits May 27, 2021 13:58

update

f19bb02

update

b6b3efa

update

b7133b9

sigmod changed the base branch from master to branch-3.1 May 27, 2021 21:18

sigmod changed the base branch from branch-3.1 to master May 27, 2021 21:18

sigmod added 3 commits May 27, 2021 14:19

Merge branch 'master' into analyzer

60fd253

update

423c7c3

add Pivot

df1e63e

add ordinal

151f996

Merge branch 'master' into analyzer

ad9dbed

sigmod added 3 commits May 28, 2021 17:18

add pruning in CTESubstitution

e54109d

merge master

5e29863

Merge branch 'master' into analyzer

0542922

sigmod changed the title ~~[WIP][SPARK-35544][SQL] Add tree pattern pruning to Analyzer rules~~ [SPARK-35544][SQL] Add tree pattern pruning to Analyzer rules May 31, 2021

update

53d07ee

hvanhovell approved these changes May 31, 2021

View reviewed changes

gengliangwang reviewed May 31, 2021

View reviewed changes

address Gengliang's comment

8252a6a

gengliangwang approved these changes Jun 1, 2021

View reviewed changes

gengliangwang closed this in 1dd0ca2 Jun 1, 2021

ulysses-you mentioned this pull request Jun 1, 2021

[SPARK-35585][SQL] Support propagate empty relation through project/filter #32724

Closed

da-tubi mentioned this pull request Sep 1, 2021

failed to show models with java.lang.AbstractMethodError eto-ai/rikai#312

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-35544][SQL] Add tree pattern pruning to Analyzer rules #32686

[SPARK-35544][SQL] Add tree pattern pruning to Analyzer rules #32686

sigmod commented May 27, 2021 •

edited

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 28, 2021

SparkQA commented May 28, 2021

SparkQA commented May 28, 2021

SparkQA commented May 29, 2021

SparkQA commented May 31, 2021

SparkQA commented May 31, 2021

sigmod commented May 31, 2021

SparkQA commented May 31, 2021

SparkQA commented May 31, 2021

SparkQA commented May 31, 2021

SparkQA commented May 31, 2021

hvanhovell left a comment

hvanhovell commented May 31, 2021

gengliangwang May 31, 2021

sigmod Jun 1, 2021

gengliangwang May 31, 2021

sigmod Jun 1, 2021

gengliangwang May 31, 2021

sigmod Jun 1, 2021

gengliangwang May 31, 2021

sigmod Jun 1, 2021

sigmod commented Jun 1, 2021

SparkQA commented Jun 1, 2021

gengliangwang commented Jun 1, 2021

SparkQA commented Jun 1, 2021

		@@ -117,6 +120,7 @@ case class AggregateExpression(
		UnresolvedAttribute(aggregateFunction.toString)
		}

[SPARK-35544][SQL] Add tree pattern pruning to Analyzer rules #32686

[SPARK-35544][SQL] Add tree pattern pruning to Analyzer rules #32686

Conversation

sigmod commented May 27, 2021 • edited

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 27, 2021

SparkQA commented May 28, 2021

SparkQA commented May 28, 2021

SparkQA commented May 28, 2021

SparkQA commented May 29, 2021

SparkQA commented May 31, 2021

SparkQA commented May 31, 2021

sigmod commented May 31, 2021

SparkQA commented May 31, 2021

SparkQA commented May 31, 2021

SparkQA commented May 31, 2021

SparkQA commented May 31, 2021

hvanhovell left a comment

Choose a reason for hiding this comment

hvanhovell commented May 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sigmod commented Jun 1, 2021

SparkQA commented Jun 1, 2021

gengliangwang commented Jun 1, 2021

SparkQA commented Jun 1, 2021

sigmod commented May 27, 2021 •

edited