[SPARK-31848][CORE][TEST] DAGSchedulerSuite: Break down the very huge test file #29747
Conversation
|
Test build #128625 has finished for PR 29747 at commit
|
|
Test build #128627 has finished for PR 29747 at commit
|
beliefer
changed the title
|
cc @Ngone51 |
| def checkAndCompleteRetryStage( | ||
| taskSetIndex: Int, | ||
| stageId: Int, | ||
| shuffleId: Int): Unit = { |
|
|
||
| class DAGSchedulerSuiteDummyException extends Exception | ||
|
|
||
| class DAGSchedulerTestHelper extends SparkFunSuite with TempLocalSparkContext with TimeLimits { |
There was a problem hiding this comment.
Offline discussion between @Ngone51 and me, decided to use DAGSchedulerTestBase
| } | ||
|
|
||
| /** A simple helper class for creating custom JobListeners */ | ||
| class SimpleListener extends JobListener { |
There was a problem hiding this comment.
This is only used by DAGSchedulerSuite. Shall we keep it at DAGSchedulerSuite?
| } | ||
|
|
||
| // Helper functions to extract commonly used code in Fetch Failure test cases | ||
| protected def setupStageAbortTest(sc: SparkContext): Unit = { |
There was a problem hiding this comment.
This is only used by DAGSchedulerSuite too. Shall we keep it at DAGSchedulerSuite?
| * @param numShufflePartitions - The number of partitions in the next stage | ||
| * @param hostNames - Host on which each task in the task set is executed | ||
| */ | ||
| protected def completeShuffleMapStageSuccessfully( |
There was a problem hiding this comment.
This is only used by DAGSchedulerSuite too. Shall we keep it at DAGSchedulerSuite? Could you check others?
There was a problem hiding this comment.
completeShuffleMapStageSuccessfully is used by DAGSchedulerSuite, BarrierExecutionSuite and IndeterminateStageSuite.
|
Test build #128706 has finished for PR 29747 at commit
|
|
Test build #128758 has started for PR 29747 at commit |
|
retest this please |
|
Test build #128786 has finished for PR 29747 at commit
|
|
cc @jiangxb1987 |
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
What changes were proposed in this pull request?
The current
DAGSchedulerSuiteis a very huge test file, we should split this suite into multiple suites which should focus on one or several major features, but not all the related behaviors.This PR will extract the following features related tests as new suite from the
DAGSchedulerSuite:Barrier execution
Indeterminate stage
Resource profile
Why are the changes needed?
Break down the very huge test files, each test suite should focus on one or several major features, but not all the related behaviors.
Does this PR introduce any user-facing change?
'No'.
How was this patch tested?
Jenkins test.