[SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec #26846
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… to ShuffleExchangeExec
|
Please help to retest. Thanks. |
|
Test build #115152 has finished for PR 26846 at commit
|
cloud-fan
reviewed
Dec 11, 2019
| @@ -64,6 +66,14 @@ case class ShuffleExchangeExec( | |||
|
|
|||
| @transient lazy val inputRDD: RDD[InternalRow] = child.execute() | |||
|
|
|||
| @transient lazy val mapOutputStatisticsFuture: Future[MapOutputStatistics] = { | |||
There was a problem hiding this comment.
can we add a comment saying that this is only needed in AQE?
|
retest this please |
cloud-fan
approved these changes
Dec 11, 2019
|
Test build #115160 has finished for PR 26846 at commit
|
dongjoon-hyun
changed the title
dongjoon-hyun
approved these changes
Dec 11, 2019
|
@cloud-fan @dongjoon-hyun add the comments. Thanks |
|
Merged to master. (The last commit is only adding a comment.) |
|
Test build #115205 has finished for PR 26846 at commit
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently
ShuffleQueryStageExeccontain the mutable status, egmapOutputStatisticsFuturevariable. So It is not easy to pass when we copyShuffleQueryStageExec. This PR will put themapOutputStatisticsFuturevariable fromShuffleQueryStageExectoShuffleExchangeExec. And then we can pass the value ofmapOutputStatisticsFuturewhen copying.Why are the changes needed?
In order to remove the mutable status in
ShuffleQueryStageExecDoes this PR introduce any user-facing change?
No
How was this patch tested?
Existing uts