Skip to content

Commit

Permalink
[SPARK-4664][Core] Throw an exception when spark.akka.frameSize > 2047
Browse files Browse the repository at this point in the history
If `spark.akka.frameSize` > 2047, it will overflow and become negative. Should have some assertion in `maxFrameSizeBytes` to warn people.

Author: zsxwing <zsxwing@gmail.com>

Closes apache#3527 from zsxwing/SPARK-4664 and squashes the following commits:

0089c7a [zsxwing] Throw an exception when spark.akka.frameSize > 2047
  • Loading branch information
zsxwing authored and rxin committed Dec 1, 2014
1 parent 6384f42 commit 1d238f2
Showing 1 changed file with 8 additions and 1 deletion.
9 changes: 8 additions & 1 deletion core/src/main/scala/org/apache/spark/util/AkkaUtils.scala
Original file line number Diff line number Diff line change
Expand Up @@ -134,9 +134,16 @@ private[spark] object AkkaUtils extends Logging {
Duration.create(conf.getLong("spark.akka.lookupTimeout", 30), "seconds")
}

private val AKKA_MAX_FRAME_SIZE_IN_MB = Int.MaxValue / 1024 / 1024

/** Returns the configured max frame size for Akka messages in bytes. */
def maxFrameSizeBytes(conf: SparkConf): Int = {
conf.getInt("spark.akka.frameSize", 10) * 1024 * 1024
val frameSizeInMB = conf.getInt("spark.akka.frameSize", 10)
if (frameSizeInMB > AKKA_MAX_FRAME_SIZE_IN_MB) {
throw new IllegalArgumentException("spark.akka.frameSize should not be greater than "
+ AKKA_MAX_FRAME_SIZE_IN_MB + "MB")
}
frameSizeInMB * 1024 * 1024
}

/** Space reserved for extra data in an Akka message besides serialized task or task result. */
Expand Down

0 comments on commit 1d238f2

Please sign in to comment.