[SPARK-38187][K8S][TESTS] Add K8S IT for `volcano` minResources cpu/memory spec #35640

Yikun · 2022-02-24T00:46:09Z

What changes were proposed in this pull request?

This PR adds two tests to make sure resource reservation supported.

Run SparkPi Jobs with minCPU
Run SparkPi Jobs with minMemory

Why are the changes needed?

Test resource reservation (min Resoruce) with volcano implementations

Does this PR introduce any user-facing change?

No, K8S IT only

How was this patch tested?

integration test

[info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (12 seconds, 738 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (13 seconds, 294 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (25 seconds, 659 milliseconds)
[info] - SPARK-38423: Run SparkPi Jobs with priorityClassName (19 seconds, 310 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (16 seconds, 467 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (29 seconds, 546 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (30 seconds, 473 milliseconds)
[info] Run completed in 2 minutes, 30 seconds.
[info] Total number of tests run: 7
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 7, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 236 s (03:56), completed 2022-3-10 9:17:46

dongjoon-hyun · 2022-02-24T21:33:17Z

docs/running-on-kubernetes.md

@@ -1356,6 +1356,26 @@ See the [configuration page](configuration.html) for information on Spark config
  </td>
  <td>3.3.0</td>
 </tr>
+<tr>
+  <td><code>spark.kubernetes.job.minCPU</code></td>


I'm just wondering if we can reuse some of the Spark configuration.

Yep, this's a good question, there were some related use case of Spark with volcano in production can be shared in here:

Case1 (this PR): minCPU = specified minCPU: for users who know their own cluster resources well, this is the basic user case. Especially, when users don't want to set minRes strictly to which the spark job real needed resource amounts, this also helps to enhance the utilization of cluster in some level when cluster resource is limited.

Case2: minCPU = driver.request + executor.number * executor.request: for users who don't care much about job resource usage.

Case3: minCPU = (driver.request + executor.number * executor.request) * factor, for users want to guarantee the resources of the job in some level, but also want to improve the utilization of the cluster.

So, we might add the basic minRes configuration first.

also cc @holdenk

We need to document what is used in case of (none), @Yikun .

If no value is specified, there will be no lower limit on job-level CPU resources.

will add it

dongjoon-hyun · 2022-02-24T21:33:26Z

docs/running-on-kubernetes.md

+  <td>3.3.0</td>
+</tr>
+<tr>
+  <td><code>spark.kubernetes.job.minMemory</code></td>


martin-g

There is a small typo in the name of method checkAnnotation

...netes/core/src/test/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStepSuite.scala

...ion-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolcanoTestsSuite.scala

dongjoon-hyun · 2022-03-03T19:46:10Z

Please check the dev mailing list . Ramping down for Apache Spark 3.3 release started, @Yikun .

https://lists.apache.org/thread/ffdk52hkvgsc5ncjh1z0nv120jowtrld

Yikun · 2022-03-04T15:33:52Z

[info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (12 seconds, 199 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (45 seconds, 453 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (43 seconds, 329 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enable) (17 seconds, 385 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enable) (37 seconds, 684 milliseconds)

...egration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolcanoSuite.scala

Yikun · 2022-03-06T07:07:30Z

[info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (12 seconds, 211 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (34 seconds, 483 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (34 seconds, 345 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enable) (17 seconds, 298 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enable) (37 seconds, 704 milliseconds)
[info] Run completed in 2 minutes, 18 seconds.
[info] Total number of tests run: 5
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 5, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

dongjoon-hyun · 2022-03-07T04:51:57Z

#35733 is merged now . Could you rebase this PR, @Yikun ?

Yikun · 2022-03-07T07:30:42Z

$ build/sbt -Pvolcano -Pkubernetes -Pkubernetes-integration-tests -Dtest.exclude.tags=minikube,r  -Dtest.include.tags=volcano  -Dspark.kubernetes.test.namespace=default "kubernetes-integration-tests/test"
[info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (12 seconds, 245 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (43 seconds, 473 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (43 seconds, 354 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enable) (18 seconds, 323 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enable) (38 seconds, 673 milliseconds)
[info] Run completed in 2 minutes, 38 seconds.
[info] Total number of tests run: 5
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 5, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 378 s (06:18), completed Mar 7, 2022 3:28:39 PM

$ k get pod
No resources found in default namespace.
$ k get queue
NAME      AGE
default   2d20h

Yikun · 2022-03-07T07:31:05Z

@dongjoon-hyun Done.

...kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala

resource-managers/kubernetes/integration-tests/src/test/resources/volcano/queue-2u-3g.yml

...ion-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolcanoTestsSuite.scala

dongjoon-hyun · 2022-03-07T08:05:53Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

@@ -299,6 +300,20 @@ private[spark] object Config extends Logging {
    .stringConf
    .createOptional

+  val KUBERNETES_JOB_MIN_CPU = ConfigBuilder("spark.kubernetes.job.minCPU")


BTW, this naming looks inconsistent to me. For the other K8s configuration, we use Core instead of CPU. Please double check this.

Yes, this might be a good suggestion for improvement. I also searched in spark, only spark.task.cpus are using cpu, and others are using cores. I guess it (cores) might be because of the name originally inherited from Yarn.

So, I think cores is also fine for me, like spark.kubernetes.job.minCores. if no objection I can change this soon.

also cc @holdenk

dongjoon-hyun

I left a few more comments.

Yikun · 2022-03-07T14:12:33Z

[info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (13 seconds, 248 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (33 seconds, 501 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (34 seconds, 336 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (17 seconds, 203 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (29 seconds, 392 milliseconds)
[info] - SPARK-38423: Run SparkPi Jobs with priorityClassName (21 seconds, 190 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (20 seconds, 239 milliseconds)
[info] Run completed in 2 minutes, 54 seconds.
[info] Total number of tests run: 7
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 7, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 362 s (06:02), completed Mar 7, 2022 10:07:21 PM

dongjoon-hyun

@Yikun
We are building PodGroup one by one manually. However, most of the configurations are static. ~~Why don't we make PodGroup implement Loadable like Pod?~~

It seems that you decided not to use volcano API for some reasons . Can we use new DefaultVolcanoClient().podGroups().load()?

Yikun · 2022-03-08T00:05:08Z

@dongjoon-hyun I'm not sure there were some misunderstanding, something below might help you understand if something confused:
We configure pod (driver and executors) one by one but we only return the PreAdditionalK8SResource once before driver creation. So there are only one podgroup is created for one job in current implementation（the pre kubernetes only calls in driver side).

If we are in the same page then:

It seems that you decided not to use volcano API for some reasons

No more hidden reason, just make sure it can be more generic, then it can be created by kubernetesClient.resourceList(preKubernetesResources: _*).createOrReplace(). So we only return volcano model in here.

Can we use new DefaultVolcanoClient().podGroups().load()?

if you meaned use this to build volcano-model, client side yaml load could be a alternative way to complete this, but I think we still need to left placeholder in yaml and do format in feature step. This perhaps not flexiable, and a little bit hard to maintain. and also introduce a volcano client module deps.

:), feel free to left any question you had, I will try my best to make work better.

dongjoon-hyun

@Yikun . Here is my PR for volcano based on my original suggestion to you. I believe the two PodGroup template configurations are enough for Volcano because it is a simpler and more extensible and future-proof way instead of introducing many new configurations one by one like this PR.

[SPARK-38455][SPARK-38187][K8S] Support driver/executor PodGroup templates #35776

…mplates ### What changes were proposed in this pull request? This PR aims to support driver/executor `PodGroup` templates like the following. ```yaml apiVersion: scheduling.volcano.sh/v1beta1 kind: PodGroup spec: minMember: 1000 minResources: cpu: "4" memory: "16Gi" priorityClassName: executor-priority queue: executor-queue ``` ### Why are the changes needed? This is a simpler, more extensible and robust way to support Volcano future because we don't need to add new configurations like #35640 for all Volcano features. ### Does this PR introduce _any_ user-facing change? No because this is a new feature. ### How was this patch tested? Pass the CIs. Closes #35776 from dongjoon-hyun/SPARK-38455. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2022-03-10T01:01:00Z

Could you rebase this PR to the master and convert it as a test case addition PR, @Yikun ?

Yikun · 2022-03-10T01:20:15Z

@dongjoon-hyun Done! Thanks! : )

Yikun · 2022-03-10T01:24:30Z

[info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (12 seconds, 738 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (13 seconds, 294 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (25 seconds, 659 milliseconds)
[info] - SPARK-38423: Run SparkPi Jobs with priorityClassName (19 seconds, 310 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (16 seconds, 467 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (29 seconds, 546 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (30 seconds, 473 milliseconds)
[info] Run completed in 2 minutes, 30 seconds.
[info] Total number of tests run: 7
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 7, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 236 s (03:56), completed 2022-3-10 9:17:46

dongjoon-hyun

+1, LGTM. Thank you, @Yikun . Merged to master.

$ build/sbt -Psparkr -Pkubernetes -Pvolcano -Pkubernetes-integration-tests -Dtest.exclude.tags=minikube -Dspark.kubernetes.test.deployMode=docker-for-desktop 'kubernetes-integration-tests/testOnly -- -z SPARK-38187'
...
[info] KubernetesSuite:
[info] VolcanoSuite:
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (24 seconds, 485 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (24 seconds, 665 milliseconds)
[info] Run completed in 1 minute, 12 seconds.
[info] Total number of tests run: 2
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 99 s (01:39), completed Mar 9, 2022 5:26:29 PM

…emory spec ### What changes were proposed in this pull request? This PR adds two tests to make sure resource reservation supported. - Run SparkPi Jobs with minCPU - Run SparkPi Jobs with minMemory ### Why are the changes needed? Test resource reservation (min Resoruce) with volcano implementations ### Does this PR introduce _any_ user-facing change? No, K8S IT only ### How was this patch tested? - integration test ``` [info] VolcanoSuite: [info] - Run SparkPi with volcano scheduler (12 seconds, 738 milliseconds) [info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (13 seconds, 294 milliseconds) [info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (25 seconds, 659 milliseconds) [info] - SPARK-38423: Run SparkPi Jobs with priorityClassName (19 seconds, 310 milliseconds) [info] - SPARK-38423: Run driver job to validate priority order (16 seconds, 467 milliseconds) [info] - SPARK-38187: Run SparkPi Jobs with minCPU (29 seconds, 546 milliseconds) [info] - SPARK-38187: Run SparkPi Jobs with minMemory (30 seconds, 473 milliseconds) [info] Run completed in 2 minutes, 30 seconds. [info] Total number of tests run: 7 [info] Suites: completed 2, aborted 0 [info] Tests: succeeded 7, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [success] Total time: 236 s (03:56), completed 2022-3-10 9:17:46 ``` Closes apache#35640 from Yikun/SPARK-38187-minRes. Authored-by: Yikun Jiang <yikunkero@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actions bot added DOCS KUBERNETES labels Feb 24, 2022

Yikun force-pushed the SPARK-38187-minRes branch from 96d7b5f to ce75f5c Compare February 24, 2022 08:39

dongjoon-hyun reviewed Feb 24, 2022

View reviewed changes

docs/running-on-kubernetes.md Outdated

<td>3.3.0</td>

</tr>

<tr>

<td><code>spark.kubernetes.job.minMemory</code></td>

Copy link

Member

dongjoon-hyun Feb 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Yikun force-pushed the SPARK-38187-minRes branch from ce75f5c to 361988f Compare March 2, 2022 13:23

martin-g reviewed Mar 2, 2022

View reviewed changes

...netes/core/src/test/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStepSuite.scala Outdated Show resolved Hide resolved

...ion-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolcanoTestsSuite.scala Outdated Show resolved Hide resolved

Yikun force-pushed the SPARK-38187-minRes branch from 361988f to 86a5f3e Compare March 4, 2022 15:32

Yikun changed the title ~~[WIP][SPARK-38187][K8S] Support resource reservation with volcano implementations~~ [SPARK-38187][K8S] Support resource reservation with volcano implementations Mar 4, 2022

Yikun marked this pull request as ready for review March 4, 2022 15:35

Yikun commented Mar 4, 2022

View reviewed changes

...egration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolcanoSuite.scala Outdated Show resolved Hide resolved

Yikun force-pushed the SPARK-38187-minRes branch 2 times, most recently from 0f42741 to d8b38a8 Compare March 5, 2022 00:44

Yikun changed the title ~~[SPARK-38187][K8S] Support resource reservation with volcano implementations~~ [SPARK-38423][K8S] Support resource reservation with volcano implementations Mar 6, 2022

Yikun changed the title ~~[SPARK-38423][K8S] Support resource reservation with volcano implementations~~ [SPARK-38187][K8S] Support resource reservation with volcano implementations Mar 6, 2022

Yikun force-pushed the SPARK-38187-minRes branch 3 times, most recently from 3f6dc20 to 8bf21e8 Compare March 6, 2022 07:06

Yikun force-pushed the SPARK-38187-minRes branch from 8bf21e8 to 05f5512 Compare March 7, 2022 07:29