[SPARK-38455][SPARK-38187][K8S] Support driver/executor `PodGroup` templates #35776

dongjoon-hyun · 2022-03-08T22:04:09Z

What changes were proposed in this pull request?

This PR aims to support driver/executor PodGroup templates like the following.

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
spec:
  minMember: 1000
  minResources:
    cpu: "4"
    memory: "16Gi"
  priorityClassName: executor-priority
  queue: executor-queue

Why are the changes needed?

This is a simpler, more extensible and robust way to support Volcano future because we don't need to add new configurations like #35640 for all Volcano features.

Does this PR introduce any user-facing change?

No because this is a new feature.

How was this patch tested?

Pass the CIs.

dongjoon-hyun · 2022-03-08T22:21:31Z

Could you review this, @viirya and @Yikun ?

viirya · 2022-03-08T22:45:44Z

...kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala

-    priorityClassName.foreach(podGroup.editOrNewSpec().withPriorityClassName(_).endSpec())
+    var spec = pg.getSpec
+    if (spec == null) spec = new PodGroupSpec
+    queue.foreach(spec.setQueue(_))


I saw there is queue in the PodGroup template. So this will overwrite it?

Yes, currently, spark.kubernetes.job.queue configuration will overwrite it. It's the behavior for non-Volcano driver/executor pod template, too.

Got it. Thanks.

viirya

Yea, I like this direction instead of adding many individual configurations.

Yikun · 2022-03-08T23:04:16Z

I take a roughly look it's a better choice to help to avoid configuration introduced. I will take a deep look and test this today.

@dongjoon-hyun Thanks!

dongjoon-hyun · 2022-03-08T23:05:03Z

Thanks, @viirya and @Yikun .

viirya · 2022-03-08T23:05:22Z

...kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala


-    priorityClassName.foreach(podGroup.editOrNewSpec().withPriorityClassName(_).endSpec())
+    var spec = pg.getSpec
+    if (spec == null) spec = new PodGroupSpec


If no template given, do we need some default pod group spec?

Yes, it's the existing code's behavior at PodGroupBuilder.

viirya

The idea looks good to me.

Yikun · 2022-03-08T23:10:11Z

...kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala

+    val template = if (kubernetesConf.isInstanceOf[KubernetesDriverConf]) {
+      kubernetesConf.get(KUBERNETES_DRIVER_PODGROUP_TEMPLATE_FILE)
+    } else {
+      kubernetesConf.get(KUBERNETES_EXECUTOR_PODGROUP_TEMPLATE_FILE)


Note that we are not support executor's separated podgroup yet, currently no ability to create pre resource for excutors in spark on k8s.

we only supported driver side podgroup or job level podgroup (share driver podgroup) now.

Ya, this is added for feature parity for your existing contribution.
getAdditionalPreKubernetesResources is used only for KubernetesDriverBuilder.

dongjoon-hyun · 2022-03-09T02:39:21Z

To fix license linter error, the licenses are added.

BTW, do you have any other concerns, @Yikun ?

Yikun · 2022-03-09T02:58:48Z

...kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala

+    var spec = pg.getSpec
+    if (spec == null) spec = new PodGroupSpec
+    queue.foreach(spec.setQueue(_))
+    priorityClassName.foreach(spec.setPriorityClassName(_))


I have only one left conern in priorityClassName about configuration overwrite priority.

Compare to pod template this might be a reasonable overwirte sort for me:

Top 1 configuration value (like queue)

Top 2 config template value (like driver/executor template)

Top 3 default value (like priorityClassName which reuse pod priority as default value)

So do you think we should change it to:

// Overwrite if queue configuration specified queue.foreach(spec.setQueue(_)) // Reuse Pod priority if priority is not set in template if (spec.getPriorityClassName == null) priorityClassName.foreach(spec.setPriorityClassName(_))

Then users can set PriorityClassName by specifying priority in template. WDYT?

I understand you are exploring all possibility. However, Apache Spark doesn't work like that. :) We prefer the user given configurations over any template files and it's simpler always. So, the current behavior is intentionally consistent with the existing one.

OK, I can understand it. This is not only exploring all possibility, but also there are many users want to specify priority more flexiable in job level.

An alternative way to help user use priority conveniently, we still keep this as your current implemtation, consider priority scheduling is very common case, do you think we could introduce a configuration like spark.kuberentes.driver.PriorityClassName to simplify config template? This also help both native default scheduler and also custom scheduler to specify spark job priority easily.

Yikun · 2022-03-09T03:01:31Z

resource-managers/kubernetes/core/src/test/resources/executor-podgroup-template.yml

+    cpu: "4"
+    memory: "16Gi"
+  priorityClassName: executor-priority
+  queue: executor-queue


nit: new line

Fixed now, @Yikun .

dongjoon-hyun · 2022-03-09T04:08:20Z

BTW, @Yikun . We can take advantage this PodGroup templates more actively in order to isolate from the other customer schedulers, the default K8s scheduler or Apache Spark itself. I'm going to propose that after this PR.

Yikun

[info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (10 seconds, 10 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (14 seconds, 304 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (13 seconds, 252 milliseconds)
[info] - SPARK-38423: Run SparkPi Jobs with priorityClassName (15 seconds, 264 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (16 seconds, 315 milliseconds)
- (coming soon)SPARK-38187: Run SparkPi Jobs with minCPU (28 seconds, 525 milliseconds)
- (coming soon)SPARK-38187: Run SparkPi Jobs with minMemory (26 seconds, 497 milliseconds)

Except the priority concern otherwise LGTM. Thanks, I also test it in my env for existing case and coming soon minRes case.

dongjoon-hyun · 2022-03-09T04:16:24Z

Thank you, @viirya and @Yikun . Merged to master for Apache Spark 3.3.0.

Yikun · 2022-03-09T04:18:36Z

We can take advantage this PodGroup templates more actively in order to isolate from the other customer schedulers, the default K8s scheduler or Apache Spark itself. I'm going to propose that after this PR.

Expected!

[SPARK-38455][K8S] Support driver/executor PodGroup templates

678dd58

github-actions bot added BUILD KUBERNETES labels Mar 8, 2022

dongjoon-hyun changed the title ~~[SPARK-38455][K8S] Support driver/executor PodGroup templates~~ [SPARK-38455][K8S] Support driver/executor PodGroup templates Mar 8, 2022

Add test prefix

fd91381

dongjoon-hyun changed the title ~~[SPARK-38455][K8S] Support driver/executor PodGroup templates~~ [SPARK-38455][SPARK-38187][K8S] Support driver/executor PodGroup templates Mar 8, 2022

viirya reviewed Mar 8, 2022

View reviewed changes

viirya approved these changes Mar 8, 2022

View reviewed changes

Yikun reviewed Mar 8, 2022

View reviewed changes

Add license

924a931

Yikun reviewed Mar 9, 2022

View reviewed changes

a

20455b7

Yikun approved these changes Mar 9, 2022

View reviewed changes

dongjoon-hyun closed this in b8c03ee Mar 9, 2022

dongjoon-hyun deleted the SPARK-38455 branch March 9, 2022 04:23

Yikun mentioned this pull request Mar 10, 2022

[SPARK-38503][K8S] Warn if VolcanoFeatureStep.getAdditionalPreKubernetesResources is used in executor side #35786

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-38455][SPARK-38187][K8S] Support driver/executor `PodGroup` templates #35776

[SPARK-38455][SPARK-38187][K8S] Support driver/executor `PodGroup` templates #35776

dongjoon-hyun commented Mar 8, 2022 •

edited

dongjoon-hyun commented Mar 8, 2022

viirya Mar 8, 2022

dongjoon-hyun Mar 8, 2022

viirya Mar 8, 2022

viirya left a comment

Yikun commented Mar 8, 2022

dongjoon-hyun commented Mar 8, 2022

viirya Mar 8, 2022

dongjoon-hyun Mar 9, 2022

viirya left a comment

Yikun Mar 8, 2022 •

edited

dongjoon-hyun Mar 9, 2022

dongjoon-hyun commented Mar 9, 2022

Yikun Mar 9, 2022 •

edited

dongjoon-hyun Mar 9, 2022

Yikun Mar 9, 2022

Yikun Mar 9, 2022

dongjoon-hyun Mar 9, 2022

dongjoon-hyun Mar 9, 2022

dongjoon-hyun commented Mar 9, 2022

Yikun left a comment

dongjoon-hyun commented Mar 9, 2022

Yikun commented Mar 9, 2022

[SPARK-38455][SPARK-38187][K8S] Support driver/executor PodGroup templates #35776

[SPARK-38455][SPARK-38187][K8S] Support driver/executor PodGroup templates #35776

Conversation

dongjoon-hyun commented Mar 8, 2022 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented Mar 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya left a comment

Choose a reason for hiding this comment

Yikun commented Mar 8, 2022

dongjoon-hyun commented Mar 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya left a comment

Choose a reason for hiding this comment

Yikun Mar 8, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Mar 9, 2022

Yikun Mar 9, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Mar 9, 2022

Yikun left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented Mar 9, 2022

Yikun commented Mar 9, 2022

[SPARK-38455][SPARK-38187][K8S] Support driver/executor `PodGroup` templates #35776

[SPARK-38455][SPARK-38187][K8S] Support driver/executor `PodGroup` templates #35776

dongjoon-hyun commented Mar 8, 2022 •

edited

Yikun Mar 8, 2022 •

edited

Yikun Mar 9, 2022 •

edited