[SPARK-10192] [core] simple test w/ failure involving a shared dependency

[squito]

just trying to increase test coverage in the scheduler, this already works. It includes a regression test for SPARK-9809

copied some test utils from #5636, we can wait till that is merged first

[SparkQA]

Test build #41475 has finished for PR 8402 at commit ccddc47.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #41487 has finished for PR 8402 at commit f768bef.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class EndListener extends SparkListener

[markhamstra]

Try this on for size. There's some more that could be done similar to this, but as an experiment I just did the refactoring for these three methods, and I think I like the results.

private implicit class ExtendedTaskSet(taskSet: TaskSet) {
    /** Validate state when creating tests for task failures. */
    def checkStageId(stageId: Int, attempt: Int): Unit = {
      assert(taskSet.stageId === stageId,
        s": expected stage $stageId, got ${taskSet.stageId}")
      assert(taskSet.stageAttemptId == attempt,
        s": expected stage attempt $attempt, got ${taskSet.stageAttemptId}")
    }

    /** Send the given CompletionEvent messages for the tasks in the TaskSet. */
    def complete(results: Seq[(TaskEndReason, Any)]) {
      assert(taskSet.tasks.length >= results.size)
      for ((result, i) <- results.zipWithIndex) {
        if (i < taskSet.tasks.length) {
          runEvent(CompletionEvent(
            taskSet.tasks(i), result._1, result._2, null, createFakeTaskInfo(), null))
        }
      }
    }

    def makeCompletions(reduceParts: Int): Seq[(Success.type, MapStatus)] = {
      taskSet.tasks.zipWithIndex.map { case (task, idx) =>
        (Success, makeMapStatus("host" + ('A' + idx).toChar, reduceParts))
      }.toSeq
    }
  }

[markhamstra]

I like this too:

def complete(results: (TaskEndReason, Any)*)

Gets rid of a lot of Seq( ... ) noise at the cost of a few : _*.

[squito]

thanks Mark -- these are the utils I'm stealing from #5636 (where these reduce a lot of noise), might be best to move this discussion there? Then again this pr is probably the easiest to merge, so maybe we can do this here first?

I don't particularly care either way, just want to avoid a fractured discussion (which would be my fault for doing this ...)

[squito]

in any case I'll take a closer look at the alternatives that you suggested here

[squito]

so after taking a closer look, these extra helper methods are only used in the completeNextStage... helpers (at least, here and in #5636). I I feel like using an implicit seems a little too heavy for that -- its a pretty minor change to the code, and it will the code harder to understand for a lot of people. I'd like to try to at least keep the tests as accessible as possible.

I agree that there are bazillion calls to complete that will benefit from varargs.

[SparkQA]

Test build #41833 has finished for PR 8402 at commit 8818b3f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class EndListener extends SparkListener

[SparkQA]

Test build #41999 has finished for PR 8402 at commit cfcf4e6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mateiz]

LGTM

[mateiz]

Although this seems to have failed another test?

[mateiz]

retest this please

[SparkQA]

Test build #42031 has finished for PR 8402 at commit cfcf4e6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

@squito can you bring this up to date?

[andrewor14]

retest this please

[SparkQA]

Test build #43990 has finished for PR 8402 at commit e93e35e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44005 has finished for PR 8402 at commit e93e35e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Merging into master and 1.6

[JoshRosen]

This appears to have broken the build. I'm going to spend 5 minutes trying to hotfix and will revert if I can't come up with a quick fix.

[JoshRosen]

Verified locally that this fixes the tests, so I'm merging to master and 1.6.