New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-36059][K8S][FOLLOWUP] Support spark.kubernetes.scheduler.name
#35499
Conversation
cc @dongjoon-hyun this may an improvement to simplify configuration on scheduler. |
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
Outdated
Show resolved
Hide resolved
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
Outdated
Show resolved
Hide resolved
spark.kubernetes.scheduler.name
spark.kubernetes.scheduler.name
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with your intention.
Please use [FOLLOWUP]
when you do the follow-up. And, in this case, please update the document together.
@dongjoon-hyun Sure, thanks, will update soon. |
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
Outdated
Show resolved
Hide resolved
Gentle ping, @Yikun . |
...rce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala
Outdated
Show resolved
Hide resolved
@@ -42,7 +42,7 @@ private[spark] abstract class KubernetesConf(val sparkConf: SparkConf) { | |||
def secretEnvNamesToKeyRefs: Map[String, String] | |||
def secretNamesToMountPaths: Map[String, String] | |||
def volumes: Seq[KubernetesVolumeSpec] | |||
def schedulerName: String | |||
def schedulerName: Option[String] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since schedulerName
is added at 3.3.0, it looks like a safe change.
@dongjoon-hyun Thanks for review. I update the test and plus doc. |
@@ -1332,21 +1332,30 @@ See the [configuration page](configuration.html) for information on Spark config | |||
<td>3.3.0</td> | |||
</tr> | |||
<tr> | |||
<td><code>spark.kubernetes.executor.scheduler.name<code></td> | |||
<td><code>spark.kubernetes.executor.scheduler.name</code></td> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
<td>(none)</td> | ||
<td> | ||
Specify the scheduler name for each executor pod. | ||
</td> | ||
<td>3.0.0</td> | ||
</tr> | ||
<tr> | ||
<td><code>spark.kubernetes.driver.scheduler.name<code></td> | ||
<td><code>spark.kubernetes.driver.scheduler.name</code></td> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What changes were proposed in this pull request?
Add
spark.kubernetes.scheduler.name
to support specify driver and executor scheduler togerther.Why are the changes needed?
Before this patch, we have to specify two configuration for driver and executor:
After this patch, we can specify executor and driver scheduler name by one configuration
Does this PR introduce any user-facing change?
Yes, a configuration added.
How was this patch tested?
UT