-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-39444][SQL] Add OptimizeSubqueries into nonExcludableRules list #36841
Conversation
cc @maryannxue and @allisonwang-db FYI |
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
Outdated
Show resolved
Hide resolved
dd81c3c
to
b8bb5d9
Compare
@@ -4,3 +4,12 @@ SELECT | |||
(SELECT min(id) FROM range(10)), | |||
(SELECT sum(id) FROM range(10)), | |||
(SELECT count(distinct id) FROM range(10)); | |||
|
|||
-- SPARK-39444 | |||
SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.Optimizer$OptimizeSubqueries; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Although it was working, I didn't realize that we allow to exclude rules whose prefix is Optimizer$...
. Is it legitimate? I'm just curious if we need to ban all exclusions for Optimizer$...
rules.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another rule is: org.apache.spark.sql.catalyst.optimizer.Optimizer$UpdateCTERelationStats
.
Merged to master. |
What changes were proposed in this pull request?
This PR adds
OptimizeSubqueries
rule into nonExcludableRules list.Why are the changes needed?
It will throw exception if user
set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.Optimizer$OptimizeSubqueries
before running this query:Exception:
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit test.