Skip to content

Commit

Permalink
[MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocati…
Browse files Browse the repository at this point in the history
…on on.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes apache#5751 from vanzin/cached-rdd-warning and squashes the following commits:

554cc07 [Marcelo Vanzin] Change message.
9efb9da [Marcelo Vanzin] [minor] [core] Warn users who try to cache RDDs with dynamic allocation on.
  • Loading branch information
Marcelo Vanzin authored and Andrew Or committed Apr 28, 2015
1 parent 53befac commit 28b1af7
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions core/src/main/scala/org/apache/spark/SparkContext.scala
Original file line number Diff line number Diff line change
Expand Up @@ -1396,6 +1396,11 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
* Register an RDD to be persisted in memory and/or disk storage
*/
private[spark] def persistRDD(rdd: RDD[_]) {
_executorAllocationManager.foreach { _ =>
logWarning(
s"Dynamic allocation currently does not support cached RDDs. Cached data for RDD " +
s"${rdd.id} will be lost when executors are removed.")
}
persistentRdds(rdd.id) = rdd
}

Expand Down

0 comments on commit 28b1af7

Please sign in to comment.