[SPARK-35874][SQL] AQE Shuffle should wait for its subqueries to finish before materializing #33058
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
cloud-fan
commented
Jun 24, 2021
| @@ -58,7 +58,11 @@ trait BroadcastExchangeLike extends Exchange { | |||
| * For registering callbacks on `relationFuture`. | |||
| * Note that calling this method may not start the execution of broadcast job. | |||
| */ | |||
| def completionFuture: scala.concurrent.Future[broadcast.Broadcast[Any]] | |||
| final def submitBroadcastJob: scala.concurrent.Future[broadcast.Broadcast[Any]] = executeQuery { | |||
There was a problem hiding this comment.
This is kind of the AQE version of "execute", as AQE won't call execute of shuffle/broadcast.
There was a problem hiding this comment.
Can we add some comments for this method?
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #140263 has finished for PR 33058 at commit
|
|
Test build #140265 has finished for PR 33058 at commit
|
cloud-fan
force-pushed
the
aqe
branch
2 times, most recently
from
00952f2
to
c964743
Compare
|
Kubernetes integration test unable to build dist. exiting with code: 1 |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #140503 has finished for PR 33058 at commit
|
…ore materializing
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Test build #140764 has finished for PR 33058 at commit
|
yaooqinn
approved these changes
Jul 8, 2021
|
LGTM |
viirya
approved these changes
Jul 8, 2021
cloud-fan
added a commit
that referenced
this pull request
Jul 8, 2021
…sh before materializing ### What changes were proposed in this pull request? Currently, AQE uses a very tricky way to trigger and wait for the subqueries: 1. submitting stage calls `QueryStageExec.materialize` 2. `QueryStageExec.materialize` calls `executeQuery` 3. `executeQuery` does some preparation works, which goes to `QueryStageExec.doPrepare` 4. `QueryStageExec.doPrepare` calls `prepare` of shuffle/broadcast, which triggers all the subqueries in this stage 5. `executeQuery` then calls `waitForSubqueries`, which does nothing because `QueryStageExec` itself has no subqueries 6. then we submit the shuffle/broadcast job, without waiting for subqueries 7. for `ShuffleExchangeExec.mapOutputStatisticsFuture`, it calls `child.execute`, which calls `executeQuery` and wait for subqueries in the query tree of `child` 8. The only missing case is: `ShuffleExchangeExec` itself may contain subqueries(repartition expression) and AQE doesn't wait for it. A simple fix would be overwriting `waitForSubqueries` in `QueryStageExec`, and forward the request to shuffle/broadcast, but this PR proposes a different and probably cleaner way: we follow `execute`/`doExecute` in `SparkPlan`, and add similar APIs in the AQE version of "execute", which gets a future from shuffle/broadcast. ### Why are the changes needed? bug fix ### Does this PR introduce _any_ user-facing change? a query fails without the fix and can run now ### How was this patch tested? new test Closes #33058 from cloud-fan/aqe. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 2df67a1) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
|
thanks for the review, merging to master/3.2! |
domybest11
pushed a commit
to domybest11/spark
that referenced
this pull request
Jun 15, 2022
…sh before materializing Currently, AQE uses a very tricky way to trigger and wait for the subqueries: 1. submitting stage calls `QueryStageExec.materialize` 2. `QueryStageExec.materialize` calls `executeQuery` 3. `executeQuery` does some preparation works, which goes to `QueryStageExec.doPrepare` 4. `QueryStageExec.doPrepare` calls `prepare` of shuffle/broadcast, which triggers all the subqueries in this stage 5. `executeQuery` then calls `waitForSubqueries`, which does nothing because `QueryStageExec` itself has no subqueries 6. then we submit the shuffle/broadcast job, without waiting for subqueries 7. for `ShuffleExchangeExec.mapOutputStatisticsFuture`, it calls `child.execute`, which calls `executeQuery` and wait for subqueries in the query tree of `child` 8. The only missing case is: `ShuffleExchangeExec` itself may contain subqueries(repartition expression) and AQE doesn't wait for it. A simple fix would be overwriting `waitForSubqueries` in `QueryStageExec`, and forward the request to shuffle/broadcast, but this PR proposes a different and probably cleaner way: we follow `execute`/`doExecute` in `SparkPlan`, and add similar APIs in the AQE version of "execute", which gets a future from shuffle/broadcast. bug fix a query fails without the fix and can run now new test Closes apache#33058 from cloud-fan/aqe. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 2df67a1) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
wangyum
pushed a commit
that referenced
this pull request
May 26, 2023
…eries to finish before materializing (#998) * [SPARK-35874][SQL] AQE Shuffle should wait for its subqueries to finish before materializing Currently, AQE uses a very tricky way to trigger and wait for the subqueries: 1. submitting stage calls `QueryStageExec.materialize` 2. `QueryStageExec.materialize` calls `executeQuery` 3. `executeQuery` does some preparation works, which goes to `QueryStageExec.doPrepare` 4. `QueryStageExec.doPrepare` calls `prepare` of shuffle/broadcast, which triggers all the subqueries in this stage 5. `executeQuery` then calls `waitForSubqueries`, which does nothing because `QueryStageExec` itself has no subqueries 6. then we submit the shuffle/broadcast job, without waiting for subqueries 7. for `ShuffleExchangeExec.mapOutputStatisticsFuture`, it calls `child.execute`, which calls `executeQuery` and wait for subqueries in the query tree of `child` 8. The only missing case is: `ShuffleExchangeExec` itself may contain subqueries(repartition expression) and AQE doesn't wait for it. A simple fix would be overwriting `waitForSubqueries` in `QueryStageExec`, and forward the request to shuffle/broadcast, but this PR proposes a different and probably cleaner way: we follow `execute`/`doExecute` in `SparkPlan`, and add similar APIs in the AQE version of "execute", which gets a future from shuffle/broadcast. bug fix a query fails without the fix and can run now new test Closes #33058 from cloud-fan/aqe. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 2df67a1) Signed-off-by: Wenchen Fan <wenchen@databricks.com> * fix ut Co-authored-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently, AQE uses a very tricky way to trigger and wait for the subqueries:
QueryStageExec.materializeQueryStageExec.materializecallsexecuteQueryexecuteQuerydoes some preparation works, which goes toQueryStageExec.doPrepareQueryStageExec.doPreparecallsprepareof shuffle/broadcast, which triggers all the subqueries in this stageexecuteQuerythen callswaitForSubqueries, which does nothing becauseQueryStageExecitself has no subqueriesShuffleExchangeExec.mapOutputStatisticsFuture, it callschild.execute, which callsexecuteQueryand wait for subqueries in the query tree ofchildShuffleExchangeExecitself may contain subqueries(repartition expression) and AQE doesn't wait for it.A simple fix would be overwriting
waitForSubqueriesinQueryStageExec, and forward the request to shuffle/broadcast, but this PR proposes a different and probably cleaner way: we followexecute/doExecuteinSparkPlan, and add similar APIs in the AQE version of "execute", which gets a future from shuffle/broadcast.Why are the changes needed?
bug fix
Does this PR introduce any user-facing change?
a query fails without the fix and can run now
How was this patch tested?
new test