Skip to content

Commit

Permalink
[SPARK-26530][CORE] Validate heartheat arguments in HeartbeatReceiver
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

Currently, heartbeat related arguments is not validated in spark, so if these args are inproperly specified, the Application may run for a while and not failed until the max executor failures reached(especially with spark.dynamicAllocation.enabled=true), thus may incurs resources waste.

This PR is to precheck these arguments in HeartbeatReceiver to fix this problem.

## How was this patch tested?

NA-just validation changes

Closes #23445 from liupc/validate-heartbeat-arguments-in-SparkSubmitArguments.

Authored-by: Liupengcheng <liupengcheng@xiaomi.com>
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
  • Loading branch information
Liupengcheng authored and Marcelo Vanzin committed Jan 24, 2019
1 parent 69dab94 commit 8d667c5
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala
Expand Up @@ -79,6 +79,15 @@ private[spark] class HeartbeatReceiver(sc: SparkContext, clock: Clock)

private val checkTimeoutIntervalMs = sc.conf.get(Network.NETWORK_TIMEOUT_INTERVAL)

private val executorHeartbeatIntervalMs = sc.conf.get(config.EXECUTOR_HEARTBEAT_INTERVAL)

require(checkTimeoutIntervalMs <= executorTimeoutMs,
s"${Network.NETWORK_TIMEOUT_INTERVAL.key} should be less than or " +
s"equal to ${config.STORAGE_BLOCKMANAGER_SLAVE_TIMEOUT.key}.")
require(executorHeartbeatIntervalMs <= executorTimeoutMs,
s"${config.EXECUTOR_HEARTBEAT_INTERVAL.key} should be less than or " +
s"equal to ${config.STORAGE_BLOCKMANAGER_SLAVE_TIMEOUT.key}")

private var timeoutCheckingTask: ScheduledFuture[_] = null

// "eventLoopThread" is used to run some pretty fast actions. The actions running in it should not
Expand Down

0 comments on commit 8d667c5

Please sign in to comment.