Skip to content

Commit

Permalink
[SPARK-16251][SPARK-20200][CORE][TEST] Flaky test: org.apache.spark.r…
Browse files Browse the repository at this point in the history
…dd.LocalCheckpointSuite.missing checkpoint block fails with informative message

## What changes were proposed in this pull request?

Currently we don't wait to confirm the removal of the block from the slave's BlockManager, if the removal takes too much time, we will fail the assertion in this test case.
The failure can be easily reproduced if we sleep for a while before we remove the block in BlockManagerSlaveEndpoint.receiveAndReply().

## How was this patch tested?
N/A

Author: Xingbo Jiang <xingbo.jiang@databricks.com>

Closes apache#18314 from jiangxb1987/LocalCheckpointSuite.
  • Loading branch information
jiangxb1987 authored and cloud-fan committed Jun 15, 2017
1 parent 1bf55e3 commit 7dc3e69
Showing 1 changed file with 8 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@

package org.apache.spark.rdd

import scala.concurrent.duration._

import org.scalatest.concurrent.Eventually.{eventually, interval, timeout}

import org.apache.spark.{LocalSparkContext, SparkContext, SparkException, SparkFunSuite}
import org.apache.spark.storage.{RDDBlockId, StorageLevel}

Expand Down Expand Up @@ -168,6 +172,10 @@ class LocalCheckpointSuite extends SparkFunSuite with LocalSparkContext {
// Collecting the RDD should now fail with an informative exception
val blockId = RDDBlockId(rdd.id, numPartitions - 1)
bmm.removeBlock(blockId)
// Wait until the block has been removed successfully.
eventually(timeout(1 seconds), interval(100 milliseconds)) {
assert(bmm.getBlockStatus(blockId).isEmpty)
}
try {
rdd.collect()
fail("Collect should have failed if local checkpoint block is removed...")
Expand Down

0 comments on commit 7dc3e69

Please sign in to comment.