Skip to content

Commit

Permalink
update
Browse files Browse the repository at this point in the history
  • Loading branch information
jiangxb1987 committed Jul 26, 2018
1 parent c16a47f commit c7600c2
Show file tree
Hide file tree
Showing 3 changed files with 14 additions and 2 deletions.
Expand Up @@ -23,6 +23,15 @@ import org.apache.spark.{Partition, TaskContext}

/**
* An RDD that applies the provided function to every partition of the parent RDD.
*
* @param prev the parent RDD.
* @param f The function used to map a tuple of (TaskContext, partition index, input iterator) to
* an output iterator.
* @param preservesPartitioning Whether the input function preserves the partitioner, which should
* be `false` unless `prev` is a pair RDD and the input function
* doesn't modify the keys.
* @param isFromBarrier Indicates whether this RDD is transformed from an RDDBarrier, a stage
* containing at least one RDDBarrier shall be turned into a barrier stage.
*/
private[spark] class MapPartitionsRDD[U: ClassTag, T: ClassTag](
var prev: RDD[T],
Expand Down
5 changes: 4 additions & 1 deletion core/src/main/scala/org/apache/spark/rdd/RDD.scala
Expand Up @@ -1853,8 +1853,11 @@ abstract class RDD[T: ClassTag](
* barrier stage.
*
* An RDD is in a barrier stage, if at least one of its parent RDD(s), or itself, are mapped from
* a RDDBarrier. This function always returns false for a [[ShuffledRDD]], since a
* an [[RDDBarrier]]. This function always returns false for a [[ShuffledRDD]], since a
* [[ShuffledRDD]] indicates start of a new stage.
*
* A [[MapPartitionsRDD]] can be transformed from an [[RDDBarrier]], under that case the
* [[MapPartitionsRDD]] shall be marked as barrier.
*/
private[spark] def isBarrier(): Boolean = isBarrier_

Expand Down
Expand Up @@ -61,7 +61,7 @@ private[spark] class ActiveJob(

var numFinished = 0

// Mark all the partitions of the stage to be not finished.
/** Resets the status of all partitions in this stage so they are marked as not finished. */
def resetAllPartitions(): Unit = {
(0 until numPartitions).foreach(finished.update(_, false))
numFinished = 0
Expand Down

0 comments on commit c7600c2

Please sign in to comment.