Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-37379][SQL] Add tree pattern pruning to CTESubstitution rule
### What changes were proposed in this pull request? This PR adds tree pattern pruning to the `CTESubstitution` analyzer rule. The rule will now exit early if the tree does not contain an `UnresolvedWith` node. ### Why are the changes needed? Analysis is eagerly performed after every DataFrame transformation. If a user's program performs a long chain of _n_ transformations to construct a large query plan then this can lead to _O(n^2)_ performance costs from `CTESubstitution` because it is applied _n_ times and each application traverses the entire logical plan tree (which contains _O(n)_ nodes). In the case of chained `withColumn` calls (leading to stacked `Project` nodes) it's possible to see _O(n^3)_ slowdowns where _n_ is the number of projects: this is because there are _n_ separate analysis phases, each of which calls `CTESubstitution.traverseAndSubstituteCTE`, where each call visits each of the _n_ `Project` nodes and each of their _O(n)_ expressions. Very large DataFrame plans typically do not use CTEs because there is not a DataFrame syntax for them (although they might appear in the plan if `sql(someQueryWithCTE)` is used). As a result, this PR's proposed optimization to skip `CTESubstitution` can greatly reduce the analysis cost for such plans. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I believe that optimizer correctness is covered by existing tests. As a toy benchmark, I ran ``` import org.apache.spark.sql.DataFrame org.apache.spark.sql.catalyst.rules.RuleExecutor.resetMetrics() (1 to 600).foldLeft(spark.range(100).toDF)((df: DataFrame, i: Int) => df.withColumn(s"col$i", $"id" % i)) println(org.apache.spark.sql.catalyst.rules.RuleExecutor.dumpTimeSpent()) ``` on my laptop before and after this PR's changes (simulating a _O(n^3)_ case). Skipping `CTESubstitution` cut the running time from ~28.4 seconds to ~15.5 seconds. The bulk of the remaining time comes from `DeduplicateRelations`, for which I plan to submit a separate optimization PR. Closes #34658 from JoshRosen/CTESubstitution-tree-pattern-pruning. Authored-by: Josh Rosen <joshrosen@databricks.com> Signed-off-by: Josh Rosen <joshrosen@databricks.com>
- Loading branch information