[SPARK-12991] [SQL] Establish a link between SparkPlan and LogicalPlan nodes #11036
Conversation
…n nodes This is a prerequisite for reusing RDDs corresponding to shared query fragments between different Spark SQL queries, which helps improve performance significantly on many analytical workloads even without explicitly caching any tables.
Add an accessor method for `_logicalPlan`.
|
Did you link to the corrected JIRA ticket? And what's your whole design? This is a big change and we need to discuss if it worth. |
|
Test build #50614 has finished for PR 11036 at commit
|
mbautin
changed the title
|
Sorry -- linked to the wrong JIRA ticket. Should be https://issues.apache.org/jira/browse/SPARK-12991. |
|
@cloud-fan : the whole design for the feature this is needed for (query fragment RDD reuse) is at https://issues.apache.org/jira/browse/SPARK-11838, but this seems to be the only part that cannot be done without modifying the Spark SQL code, because we need to find logical plans corresponding to generated RDDs somehow. |
|
This is not going to work at all with whole-stage codgen, in which we collapse all pipelinable operators into a single generated function. |
|
Even in that case, we could still obtain RDDs corresponding to SparkPlan nodes at stage boundaries, right? We would still find that useful in our query workload. |
|
How would the change here help you with that? |
|
Test build #51606 has finished for PR 11036 at commit
|
This is a prerequisite for reusing RDDs corresponding to shared query
fragments between different Spark SQL queries, which helps improve
performance significantly on many analytical workloads even without
explicitly caching any tables.