Skip to content

Commit

Permalink
[SPARK-16251][SPARK-20200][CORE][TEST] Flaky test: org.apache.spark.r…
Browse files Browse the repository at this point in the history
…dd.LocalCheckpointSuite.missing checkpoint block fails with informative message

## What changes were proposed in this pull request?

Currently we don't wait to confirm the removal of the block from the slave's BlockManager, if the removal takes too much time, we will fail the assertion in this test case.
The failure can be easily reproduced if we sleep for a while before we remove the block in BlockManagerSlaveEndpoint.receiveAndReply().

## How was this patch tested?
N/A

Author: Xingbo Jiang <xingbo.jiang@databricks.com>

Closes #18314 from jiangxb1987/LocalCheckpointSuite.

(cherry picked from commit 7dc3e69)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
jiangxb1987 authored and cloud-fan committed Jun 15, 2017
1 parent b5504f6 commit 76ee41f
Showing 1 changed file with 8 additions and 0 deletions.
Expand Up @@ -17,6 +17,10 @@

package org.apache.spark.rdd

import scala.concurrent.duration._

import org.scalatest.concurrent.Eventually.{eventually, interval, timeout}

import org.apache.spark.{LocalSparkContext, SparkContext, SparkException, SparkFunSuite}
import org.apache.spark.storage.{RDDBlockId, StorageLevel}

Expand Down Expand Up @@ -168,6 +172,10 @@ class LocalCheckpointSuite extends SparkFunSuite with LocalSparkContext {
// Collecting the RDD should now fail with an informative exception
val blockId = RDDBlockId(rdd.id, numPartitions - 1)
bmm.removeBlock(blockId)
// Wait until the block has been removed successfully.
eventually(timeout(1 seconds), interval(100 milliseconds)) {
assert(bmm.getBlockStatus(blockId).isEmpty)
}
try {
rdd.collect()
fail("Collect should have failed if local checkpoint block is removed...")
Expand Down

0 comments on commit 76ee41f

Please sign in to comment.