[SPARK-45009][SQL] Decorrelate predicate subqueries in join condition #42725
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andylam-db
changed the title
andylam-db
changed the title
jchen5
reviewed
Aug 29, 2023
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
Outdated
Show resolved
Hide resolved
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
Outdated
Show resolved
Hide resolved
jchen5
reviewed
Aug 30, 2023
There was a problem hiding this comment.
Let's also add test cases for other join types: right, full, semi, anti
Looks pretty good overall! CC @agubichev
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
Outdated
Show resolved
Hide resolved
agubichev
approved these changes
Sep 1, 2023
agubichev
approved these changes
Sep 5, 2023
andylam-db
commented
Sep 6, 2023
| @@ -55,7 +55,8 @@ abstract class Optimizer(catalogManager: CatalogManager) | |||
| Set( | |||
| "PartitionPruning", | |||
| "RewriteSubquery", | |||
| "Extract Python UDFs") | |||
| "Extract Python UDFs", | |||
| "Infer Filters") | |||
There was a problem hiding this comment.
"Infer Filters" is not inherently idempotent. We need to exclude it from the idempotency check so that tests will pass.
|
Looking for reviews! @cloud-fan @sigmod |
jchen5
reviewed
Sep 6, 2023
jchen5
approved these changes
Sep 6, 2023
|
Pinging for reviews! @allisonwang-db @cloud-fan |
allisonwang-db
approved these changes
Oct 5, 2023
cloud-fan
reviewed
Oct 11, 2023
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
Show resolved
Hide resolved
cloud-fan
reviewed
Oct 11, 2023
jchen5
reviewed
Oct 11, 2023
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
Outdated
Show resolved
Hide resolved
cloud-fan
approved these changes
Oct 12, 2023
|
@cloud-fan I think the build is failing because of unrelated failing tests -- timeouts for 1: 2: |
|
yea they are unrelated, thanks, merging to master! |
gengliangwang
pushed a commit
that referenced
this pull request
May 23, 2024
…tion of predicate subqueries in join condition which reference both join child ### What changes were proposed in this pull request? This is a follow up PR for #42725, which decorrelates predicate subqueries in join conditions. I forgot to add the error class definition for the case where the subquery references both join children, and test cases for it. ### Why are the changes needed? To show a clear error message when the condition is hit. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added SQL test and golden files. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46708 from andylam-db/follow-up-decorrelate-subqueries-in-join-cond. Authored-by: Andy Lam <andy.lam@databricks.com> Signed-off-by: Gengliang Wang <gengliang@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Pulling up correlated subquery predicates in Joins, and re-writing them into ExistenceJoins if they are not pushed down into the join inputs.
Why are the changes needed?
This change allows correlated IN and EXISTS subqueries in join condition. This is valid SQL that is not yet supported by Spark SQL.
Does this PR introduce any user-facing change?
Yes, previously unsupported queries become supported.
How was this patch tested?
Added SQL tests for IN and EXISTS in join conditions, and crossed-check correctness with postgres (except for ANTI joins, which are not supported in postgres).
Permutations of the tests: