Skip to content

Commit

Permalink
[SPARK-10546] Check partitionId's range in ExternalSorter#spill()
Browse files Browse the repository at this point in the history
See this thread for background:
http://search-hadoop.com/m/q3RTt0rWvIkHAE81

We should check the range of partition Id and provide meaningful message through exception.

Alternatively, we can use abs() and modulo to force the partition Id into legitimate range. However, expectation is that user should correct the logic error in his / her code.

Author: tedyu <yuzhihong@gmail.com>

Closes #8703 from tedyu/master.
  • Loading branch information
tedyu authored and srowen committed Sep 11, 2015
1 parent 5f46444 commit b231ab8
Showing 1 changed file with 2 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,8 @@ private[spark] class ExternalSorter[K, V, C](
val it = collection.destructiveSortedWritablePartitionedIterator(comparator)
while (it.hasNext) {
val partitionId = it.nextPartition()
require(partitionId >= 0 && partitionId < numPartitions,
s"partition Id: ${partitionId} should be in the range [0, ${numPartitions})")
it.writeNext(writer)
elementsPerPartition(partitionId) += 1
objectsWritten += 1
Expand Down

0 comments on commit b231ab8

Please sign in to comment.