[SPARK-5316] [CORE] DAGScheduler may make shuffleToMapStage leak if getParentStages failes

[YanTangZhai]

DAGScheduler may make shuffleToMapStage leak if getParentStages failes.
If getParentStages has exception for example input path does not exist, DAGScheduler would fail to handle job submission, while shuffleToMapStage may be put some records when getParentStages. However these records in shuffleToMapStage aren't going to be cleaned.
A simple job as follows:

val inputFile1 = ... // Input path does not exist when this job submits
val inputFile2 = ...
val outputFile = ...
val conf = new SparkConf()
val sc = new SparkContext(conf)
val rdd1 = sc.textFile(inputFile1)
                    .flatMap(line => line.split(" "))
                    .map(word => (word, 1))
                    .reduceByKey(_ + _, 1)
val rdd2 = sc.textFile(inputFile2)
                    .flatMap(line => line.split(","))
                    .map(word => (word, 1))
                    .reduceByKey(_ + _, 1)
try {
  val rdd3 = new PairRDDFunctions(rdd1).join(rdd2, 1)
  rdd3.saveAsTextFile(outputFile)
} catch {
  case e : Exception =>
      logError(e)
}
// print the information of DAGScheduler's shuffleToMapStage to check
// whether it still has uncleaned records.
...

[SparkQA]

Test build #25759 has finished for PR 4105 at commit 50291ca.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

This is reasonable, but it doesn't handle Error. I don't think you need a new 'inner' method? it also duplicates the job cleanup code.

[andrewor14]

@YanTangZhai When will getParentStages throw an exception though? If something like an input path doesn't exist, then we propagate the exception back to the caller anyway so the job won't succeed. I think we should just let the scheduler handle this case of failure for simplicity. I would prefer to close this PR since I don't think it buys us much for the complexity it introduces.