[SPARK-35545][SQL] Split SubqueryExpression's children field into outer attributes and join conditions #32687
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Kubernetes integration test unable to build dist. exiting with code: 1 |
|
cc @cloud-fan |
|
Test build #139036 has finished for PR 32687 at commit
|
cloud-fan
reviewed
May 28, 2021
| @@ -62,11 +62,13 @@ abstract class PlanExpression[T <: QueryPlan[_]] extends Expression { | |||
| */ | |||
There was a problem hiding this comment.
can we add classdoc to explain these parameters?
cloud-fan
approved these changes
May 28, 2021
|
Kubernetes integration test unable to build dist. exiting with code: 1 |
|
Test build #139071 has finished for PR 32687 at commit
|
|
thanks, merging to master! |
MaxGekk
pushed a commit
that referenced
this pull request
Jun 21, 2021
…queryExpression refactor ### What changes were proposed in this pull request? Add a test. ### Why are the changes needed? The SubqueryExpression refactor PR #32687 actually fixes the bug of `SubqueryExpression.references`. So this follow-up PR adds a regression unit test for it. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added a new test. Closes #32990 from Ngone51/spark-35545-followup. Authored-by: yi.wu <yi.wu@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR refactors
SubqueryExpressionclass. It removes the children field from SubqueryExpression's constructor and addsouterAttrsandjoinCond.Why are the changes needed?
Currently, the children field of a subquery expression is used to store both collected outer references in the subquery plan and join conditions after correlated predicates are pulled up.
For example:
SELECT (SELECT max(c1) FROM t1 WHERE t1.c1 = t2.c1) FROM t2During the analysis phase, outer references in the subquery are stored in the children field:
scalar-subquery [t2.c1], but after the optimizer rulePullupCorrelatedPredicates, the children field will be used to store the join conditions, which contain both the inner and the outer references:scalar-subquery [t1.c1 = t2.c1]. This is why the references of SubqueryExpression excludes the inner plan's output:spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala
Lines 68 to 69 in 29ed1a2
This can be confusing and error-prone. The references for a subquery expression should always be defined as outer attribute references.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing tests.