[SPARK-32875][CORE][TEST] TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract the general method. #29754

beliefer · 2020-09-15T04:08:37Z

What changes were proposed in this pull request?

TaskSchedulerImplSuite always check the results show below:

   val zeroCoreWorkerOffers = IndexedSeq(new WorkerOffer("executor0", "host0", 0),
      new WorkerOffer("executor1", "host1", 0))
    val taskSet = FakeTask.createTaskSet(1)
    taskScheduler.submitTasks(taskSet)
    var taskDescriptions = taskScheduler.resourceOffers(zeroCoreWorkerOffers).flatten
    assert(0 === taskDescriptions.length)

We can extract it as a generic method.

Why are the changes needed?

Extract a generic method.

Does this PR introduce any user-facing change?

'No'.

How was this patch tested?

Jenkins test

SparkQA · 2020-09-15T07:05:03Z

Test build #128694 has finished for PR 29754 at commit 4240990.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

beliefer · 2020-09-15T07:46:21Z

retest this please

SparkQA · 2020-09-15T10:28:41Z

Test build #128705 has finished for PR 29754 at commit 4240990.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

beliefer · 2020-09-16T09:03:51Z

retest this please

SparkQA · 2020-09-16T12:14:57Z

Test build #128759 has finished for PR 29754 at commit 4240990.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Ngone51 · 2020-09-22T04:34:33Z

core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala

+  private def submitTasksAndCheck(
+      offers: IndexedSeq[WorkerOffer],
+      taskSets: Seq[TaskSet] = Seq(FakeTask.createTaskSet(1)),
+      taskSchedulerOpt: Option[TaskSchedulerImpl] = None) (f: Seq[TaskDescription] => Any): Any = {


We can use the global taskScheduler as the default value?

Ngone51 · 2020-09-22T04:35:56Z

core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala

@@ -140,6 +140,16 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with B
    taskScheduler
  }

+  private def submitTasksAndCheck(
+      offers: IndexedSeq[WorkerOffer],


I found that many tests use the same WorkerOffer, e.g., WorkerOffer("executor0", "host0", 1). Shall we make it as the default value as well?

Ngone51 · 2020-09-22T04:41:14Z

core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala

-    assert(taskDescriptions.map(_.index).sorted == Seq(0, 1))
-    assert(manager.copiesRunning.take(2) === Array(1, 1))
+    submitTasksAndCheck(IndexedSeq(WorkerOffer("executor2", "host2", 1),
+      WorkerOffer("executor3", "host3", 1)), Seq.empty) { taskDescriptions =>


Personally, I feel it's weird that we're submitting empty task sets here... submitTasksAndCheck maybe not appropriate for this case? Or you could redesign the submitTasksAndCheck?

OK. Let's not change this matter.

SparkQA · 2020-09-22T10:06:54Z

Test build #128971 has finished for PR 29754 at commit 57e8eb6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

beliefer · 2020-10-26T11:05:17Z

cc @dongjoon-hyun

…fers-assert

SparkQA · 2020-11-30T10:38:29Z

Test build #131975 has finished for PR 29754 at commit 15c63c3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

beliefer · 2020-12-03T09:02:54Z

retest this please

SparkQA · 2020-12-03T10:08:34Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36711/

SparkQA · 2020-12-03T10:36:11Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36711/

SparkQA · 2020-12-03T11:41:41Z

Test build #132112 has finished for PR 29754 at commit 15c63c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

beliefer · 2020-12-10T01:56:44Z

cc @jiangxb1987

github-actions · 2021-03-22T00:49:37Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

beliefer · 2021-03-22T03:05:59Z

cc @jiangxb1987

dongjoon-hyun · 2021-03-23T01:26:43Z

Since the author (@beliefer ) is active, I removed the Stale tag and reopen this PR.

@beliefer , could you resolve the conflicts?

beliefer · 2021-03-23T02:23:33Z

@beliefer , could you resolve the conflicts?
@dongido001 Thanks!

…fers-assert

SparkQA · 2021-03-23T03:08:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40959/

SparkQA · 2021-03-23T03:18:41Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40959/

SparkQA · 2021-03-23T04:23:17Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40962/

SparkQA · 2021-03-23T04:55:36Z

Test build #136373 has finished for PR 29754 at commit 15c63c3.

This patch fails SparkR unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2021-03-23T04:56:45Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40962/

SparkQA · 2021-03-23T05:49:43Z

Test build #136378 has finished for PR 29754 at commit f5229e6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

beliefer · 2021-03-23T10:27:47Z

cc @jiangxb1987

github-actions · 2021-07-02T00:07:25Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

beliefer and others added 18 commits June 19, 2020 10:36

Reuse completeNextStageWithFetchFailure

4a6f903

Merge remote-tracking branch 'upstream/master'

96456e2

Merge remote-tracking branch 'upstream/master'

4314005

Merge remote-tracking branch 'upstream/master'

d6af4a7

Merge remote-tracking branch 'upstream/master'

f69094f

Merge remote-tracking branch 'upstream/master'

b86a42d

Merge branch 'master' of github.com:beliefer/spark

2ac5159

Merge remote-tracking branch 'upstream/master'

9021d6c

Merge branch 'master' of github.com:beliefer/spark

74a2ef4

Merge remote-tracking branch 'upstream/master'

9828158

Merge remote-tracking branch 'upstream/master'

9cd1aaf

Merge remote-tracking branch 'upstream/master'

abfcbb9

Merge remote-tracking branch 'upstream/master'

07c6c81

Merge remote-tracking branch 'upstream/master'

580130b

Merge branch 'master' of github.com:beliefer/spark

3712808

Merge remote-tracking branch 'upstream/master'

6107413

Merge remote-tracking branch 'upstream/master'

4b799b4

Extract pattern for submitTasks + resourceOffers + assert

4240990

probot-autolabeler bot added the CORE label Sep 15, 2020

Ngone51 reviewed Sep 22, 2020

View reviewed changes

Optimize code.

57e8eb6

Merge branch 'master' into extract-pattern-for-submitTasks-resourceOf…

15c63c3

…fers-assert

github-actions bot added the Stale label Mar 22, 2021

github-actions bot closed this Mar 23, 2021

dongjoon-hyun removed the Stale label Mar 23, 2021

dongjoon-hyun reopened this Mar 23, 2021

Merge branch 'master' into extract-pattern-for-submitTasks-resourceOf…

f5229e6

…fers-assert

github-actions bot added the Stale label Jul 2, 2021

github-actions bot closed this Jul 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32875][CORE][TEST] TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract the general method. #29754

[SPARK-32875][CORE][TEST] TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract the general method. #29754

beliefer commented Sep 15, 2020

SparkQA commented Sep 15, 2020

beliefer commented Sep 15, 2020

SparkQA commented Sep 15, 2020

beliefer commented Sep 16, 2020

SparkQA commented Sep 16, 2020

Ngone51 Sep 22, 2020

beliefer Sep 22, 2020

Ngone51 Sep 22, 2020

beliefer Sep 22, 2020

Ngone51 Sep 22, 2020

beliefer Sep 22, 2020

SparkQA commented Sep 22, 2020

beliefer commented Oct 26, 2020

SparkQA commented Nov 30, 2020

beliefer commented Dec 3, 2020

SparkQA commented Dec 3, 2020

SparkQA commented Dec 3, 2020

SparkQA commented Dec 3, 2020

beliefer commented Dec 10, 2020

github-actions bot commented Mar 22, 2021

beliefer commented Mar 22, 2021

dongjoon-hyun commented Mar 23, 2021

beliefer commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

beliefer commented Mar 23, 2021

github-actions bot commented Jul 2, 2021

[SPARK-32875][CORE][TEST] TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract the general method. #29754

[SPARK-32875][CORE][TEST] TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract the general method. #29754

Conversation

beliefer commented Sep 15, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Sep 15, 2020

beliefer commented Sep 15, 2020

SparkQA commented Sep 15, 2020

beliefer commented Sep 16, 2020

SparkQA commented Sep 16, 2020

Ngone51 Sep 22, 2020

Choose a reason for hiding this comment

beliefer Sep 22, 2020

Choose a reason for hiding this comment

Ngone51 Sep 22, 2020

Choose a reason for hiding this comment

beliefer Sep 22, 2020

Choose a reason for hiding this comment

Ngone51 Sep 22, 2020

Choose a reason for hiding this comment

beliefer Sep 22, 2020

Choose a reason for hiding this comment

SparkQA commented Sep 22, 2020

beliefer commented Oct 26, 2020

SparkQA commented Nov 30, 2020

beliefer commented Dec 3, 2020

SparkQA commented Dec 3, 2020

SparkQA commented Dec 3, 2020

SparkQA commented Dec 3, 2020

beliefer commented Dec 10, 2020

github-actions bot commented Mar 22, 2021

beliefer commented Mar 22, 2021

dongjoon-hyun commented Mar 23, 2021

beliefer commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

beliefer commented Mar 23, 2021

github-actions bot commented Jul 2, 2021