[WIP][SPARK-35677][Core][SQL] Support dynamic range of executor numbers for dynamic allocation #32819
Conversation
|
Test build #139470 has finished for PR 32819 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
yaooqinn
changed the title
|
Kubernetes integration test unable to build dist. exiting with code: 1 |
|
Test build #139474 has finished for PR 32819 at commit
|
|
Kubernetes integration test unable to build dist. exiting with code: 1 |
|
Test build #139496 has finished for PR 32819 at commit
|
|
cc @cloud-fan @wangyum thanks |
|
retest this please |
|
Test build #139519 has finished for PR 32819 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
| @@ -80,6 +82,22 @@ case class SetCommand(kv: Option[(String, Option[String])]) | |||
| } | |||
| (keyValueOutput, runFunc) | |||
|
|
|||
| case Some((DYN_ALLOCATION_MIN_EXECUTORS.key, Some(value))) => | |||
There was a problem hiding this comment.
This is an administration operation. I am not sure it's a good idea to control this by just run set command for long running service such as thrift server as users can easily run this command, and misuse this feature.
How about expose this via the Rest API?
| event: SparkListenerExecutorAllocatorRangeUpdate): Unit = { | ||
| val newLower = event.lower.getOrElse(_minNumExecutors) | ||
| val newUpper = event.upper.getOrElse(_maxNumExecutors) | ||
| validateSettings(newLower, newUpper) |
There was a problem hiding this comment.
I believe this should be like
try {
validateSettings(newLower, newUpper)
_minNumExecutors = newLower
_maxNumExecutors = newUpper
} catch {
....
}
Otherwise exception is thrown in this listener.
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
What changes were proposed in this pull request?
Currently, Spark allows users to set scalability within a Spark application using dynamic allocation.
spark.dynamicAllocation.minExecutors&spark.dynamicAllocation.maxExecutorsare used for scaling up and down. Within an application，Spark tactfully use them to request executors from cluster manager according to the real-time workload. Once set, the range is fixed through the whole application lifecycle. This is not very convenient for long-running application when the range should be changeable for some cases, such as:Why are the changes needed?
make the dynamic allocation for long term Spark applications
Does this PR introduce any user-facing change?
Configs below are changeable:
spark.dynamicAllocation.maxExecutors
spark.dynamicAllocation.minExecutors
How was this patch tested?
new tests