[SPARK-16958] [SQL] Reuse subqueries within the same query

[davies]

What changes were proposed in this pull request?

There could be multiple subqueries that generate same results, we could re-use the result instead of running it multiple times.

This PR also cleanup up how we run subqueries.

For SQL query

select id,(select avg(id) from t) from t where id > (select avg(id) from t)

The explain is

== Physical Plan ==
*Project [id#15L, Subquery subquery29 AS scalarsubquery()#35]
:  +- Subquery subquery29
:     +- *HashAggregate(keys=[], functions=[avg(id#15L)])
:        +- Exchange SinglePartition
:           +- *HashAggregate(keys=[], functions=[partial_avg(id#15L)])
:              +- *Range (0, 1000, splits=4)
+- *Filter (cast(id#15L as double) > Subquery subquery29)
   :  +- Subquery subquery29
   :     +- *HashAggregate(keys=[], functions=[avg(id#15L)])
   :        +- Exchange SinglePartition
   :           +- *HashAggregate(keys=[], functions=[partial_avg(id#15L)])
   :              +- *Range (0, 1000, splits=4)
   +- *Range (0, 1000, splits=4)

The visualized plan:

How was this patch tested?

Existing tests.

[SparkQA]

Test build #63389 has finished for PR 14548 at commit 1348ba7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • trait ExecSubqueryExpression extends SubqueryExpression
  • case class InSubquery(
  • case class ReuseSubquery(conf: SQLConf) extends Rule[SparkPlan]

[hvanhovell]

A large part of this class is shared with BroadcastExchangeExec. Should we try to factor out common functionality?

[davies]

I think it's OK to have some duplicated code here, over abstracted code is actually harder to read.

[hvanhovell]

@davies this looks pretty good. I am very excited about the SparkPlan clean-up!

[davies]

@hvanhovell Had posted an picture, check it out.

[SparkQA]

Test build #63560 has finished for PR 14548 at commit 8444447.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63563 has finished for PR 14548 at commit dd1581b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

Cool picture!

[hvanhovell]

LGTM

[davies]

Merging it into master, thanks!

[JkSelf]

@davies @hvanhovell @gatorsmile
Here the subquery reuse may be does not work. In my test, I found the visualized plan do show the subquery is executed once as following.

But in deed, the stage of same subquery execute maybe not once as following:

Maybe I miss some knowledge, can you help verify this? Thanks for your help!

[hvanhovell]

@JkSelf can you file a JIRA ticket?

[JkSelf]

@hvanhovell , Thanks for your help and I have filed Jira 26639.