[SPARK-33261][K8S] Add a developer API for custom feature steps #30206

holdenk · 2020-10-30T17:30:49Z

What changes were proposed in this pull request?

Add a developer API for custom driver & executor feature steps.

Why are the changes needed?

While we allow templates for the basis of pod creation, some deployments need more flexibility in how the pods are configured. This adds a developer API for custom deployments.

Does this PR introduce any user-facing change?

New developer API.

How was this patch tested?

Extended tests to verify custom step is applied when configured.

SparkQA · 2020-10-30T17:54:41Z

Test build #130463 has finished for PR 30206 at commit 4838f42.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait KubernetesFeatureConfigStep

dongjoon-hyun

Thanks, @holdenk . Can we make this accept multiple steps as a comma separated style?

cc @jiangzho and @aaruna

SparkQA · 2020-10-30T18:23:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35069/

SparkQA · 2020-10-30T18:44:07Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35069/

holdenk · 2020-10-30T19:26:05Z

That's a great suggestion @dongjoon-hyun , I'll try and get that done today :)

SparkQA · 2020-10-30T22:36:07Z

Test build #130477 has finished for PR 30206 at commit 09febce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-30T23:07:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35081/

SparkQA · 2020-10-30T23:35:38Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35081/

...ernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilder.scala

...s/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBuilder.scala

...es/core/src/test/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilderSuite.scala

...ce-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/PodBuilderSuite.scala

dongjoon-hyun

Thank you so much for the updates. It's great. I left a few minor comments about naming and a new test case.

...s/core/src/main/scala/org/apache/spark/deploy/k8s/features/KubernetesFeatureConfigStep.scala

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

tgravescs · 2020-11-02T14:30:48Z

a few other things is that SparkPod is private[spark] as well.

This I think opens it up for user to do anything with the Pod, correct me if I'm wrong, which is good and possibly bad. Are there any security concerns here? Like seeing secrets they shouldn't be able to. I can't think of any in the Spark ones directly but wanted to ask more expert in the k8s area.

I'm curious, what sort of thing can't be done via templates?

holdenk · 2020-11-03T17:43:01Z

So since we allow the user to specify templates anyways I don't see this as opening any additional security concerns. As for why templates are not enough, for some deployments I believe people want to be able to customize the pod at the end of the feature steps rather than at the start. I think also there are situations where we need to generate stuff based on pod name.

holdenk · 2020-12-03T00:04:18Z

Let's open up SparkPod to an Experimental dev API. WDYT @tgravescs ?

tgravescs · 2020-12-03T15:04:33Z

yeah that seems fine to me to make SparkPod developer api

dongjoon-hyun · 2020-12-06T09:27:03Z

Since we have an agreement, could you resolve the conflict to proceed further, @holdenk ?

Also, please update the PR according to the existing comments.

The comment is outdated.

dongjoon-hyun · 2020-12-09T19:30:52Z

Thank you so much for your update, @holdenk !

tgravescs · 2020-12-09T19:43:03Z

is this targeted for 3.1?

dongjoon-hyun · 2020-12-09T19:44:29Z

@tgravescs . I believe so because this is a part of K8s GA in Apache Spark 3.1.

dongjoon-hyun · 2020-12-09T19:45:23Z

Please let us know if you have any concerns on this. We can postpone it to Apache Spark 3.2 if you want to do so.

SparkQA · 2020-12-09T19:47:06Z

Test build #132510 has finished for PR 30206 at commit a66c00f.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkPod.scala

SparkQA · 2020-12-11T23:06:53Z

Test build #132679 has finished for PR 30206 at commit e990f33.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-11T23:37:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37282/

SparkQA · 2020-12-12T00:00:58Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37282/

SparkQA · 2020-12-12T01:05:53Z

Test build #132684 has finished for PR 30206 at commit cfe5bc1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class TestStepTwo extends KubernetesFeatureConfigStep

SparkQA · 2020-12-12T01:36:54Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37287/

SparkQA · 2020-12-12T02:00:32Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37287/

holdenk · 2020-12-14T19:38:53Z

Merging to the current dev branch for 3.2.

### What changes were proposed in this pull request? Add a developer API for custom driver & executor feature steps. ### Why are the changes needed? While we allow templates for the basis of pod creation, some deployments need more flexibility in how the pods are configured. This adds a developer API for custom deployments. ### Does this PR introduce _any_ user-facing change? New developer API. ### How was this patch tested? Extended tests to verify custom step is applied when configured. Closes apache#30206 from holdenk/SPARK-33261-allow-people-to-extend-pod-feature-steps. Authored-by: Holden Karau <hkarau@apple.com> Signed-off-by: Holden Karau <hkarau@apple.com>

…Step developer api ### What changes were proposed in this pull request? This patch adds the support for extending user feature steps with configuration by adding 2 developer api: - `KubernetesDriverCustomFeatureConfigStep`: to help user extend custom feature step in executor side - `KubernetesExecutorCustomFeatureConfigStep`: to help user extend custom feature step in driver side Before this patch user can only add feature step like: - `class TestStep extends KubernetesFeatureConfigStep`: without any kubernetes conf After this patch user can add feature step with configuration like: - `class TestStepWithDriverConf extends KubernetesDriverCustomFeatureConfigStep`: only driver - `class TestStepWithExecConf extends KubernetesExecutorCustomFeatureConfigStep`: only executor - `class TestStepWithK8SConf extends KubernetesDriverCustomFeatureConfigStep with KubernetesExecutorCustomFeatureConfigStep`: both driver and executor ### Why are the changes needed? In #30206 , a developer API for custom feature steps has been added, but it didn't support initialize user feature step with kubernetes conf (like `KubernetesConf`/`KubernetesDriverConf`/`KubernetesExecutorConf`). In most of scenarios, users want to make corresponding changes in their feature steps according to the configuration. Such as, the customized scheduler scenario, user wants to configure pod according to passed job configuration. ### Does this PR introduce _any_ user-facing change? Improve the developer API for for custom feature steps. ### How was this patch tested? - Added UT - Runing k8s integration test manaully: `build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test` Closes: #34924 Closes #35345 from Yikun/SPARK-37145-alt. Lead-authored-by: Yikun Jiang <yikunkero@gmail.com> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…cludedFeatureSteps` ### What changes were proposed in this pull request? This PR aims to support `spark.kubernetes.(driver|executor).pod.excludedFeatureSteps` configuration. ### Why are the changes needed? Since Apache Spark 3.2, we have been providing `spark.kubernetes.(driver|executor).pod.featureSteps`. - #30206 This PR aims to allow users to exclude feature steps selectively by configurations. Please note that this is designed to allow to exclude all steps including both built-in and user-provided steps. ### Does this PR introduce _any_ user-facing change? No because this is a new feature. ### How was this patch tested? Pass the CIs with newly added test cases. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #51522 from dongjoon-hyun/SPARK-52830. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…cludedFeatureSteps` ### What changes were proposed in this pull request? This PR aims to support `spark.kubernetes.(driver|executor).pod.excludedFeatureSteps` configuration. ### Why are the changes needed? Since Apache Spark 3.2, we have been providing `spark.kubernetes.(driver|executor).pod.featureSteps`. - apache#30206 This PR aims to allow users to exclude feature steps selectively by configurations. Please note that this is designed to allow to exclude all steps including both built-in and user-provided steps. ### Does this PR introduce _any_ user-facing change? No because this is a new feature. ### How was this patch tested? Pass the CIs with newly added test cases. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#51522 from dongjoon-hyun/SPARK-52830. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun changed the title ~~[SPARK-33261][KUBERNETES] Add a developer API for custom feature steps.~~ [SPARK-33261][K8S] Add a developer API for custom feature steps Oct 30, 2020

dongjoon-hyun requested changes Oct 30, 2020

View reviewed changes

dongjoon-hyun reviewed Oct 31, 2020

View reviewed changes

...ernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilder.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Oct 31, 2020

View reviewed changes

...s/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBuilder.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Oct 31, 2020

View reviewed changes

...es/core/src/test/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilderSuite.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Oct 31, 2020

View reviewed changes

...ce-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/PodBuilderSuite.scala Show resolved Hide resolved

dongjoon-hyun previously requested changes Oct 31, 2020

View reviewed changes

HyukjinKwon reviewed Nov 1, 2020

View reviewed changes

...s/core/src/main/scala/org/apache/spark/deploy/k8s/features/KubernetesFeatureConfigStep.scala Show resolved Hide resolved

tgravescs reviewed Nov 2, 2020

View reviewed changes

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated Show resolved Hide resolved

holdenk force-pushed the SPARK-33261-allow-people-to-extend-pod-feature-steps branch from 09febce to 2117566 Compare December 9, 2020 19:22

github-actions bot added the KUBERNETES label Dec 9, 2020

dongjoon-hyun reviewed Dec 9, 2020

View reviewed changes

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkPod.scala Outdated Show resolved Hide resolved

holdenk added 12 commits December 11, 2020 14:49

Allow for multiple steps

7f35eb9

Expose the SparkPod as dev API

e0edd51

Also unstable

ea04937

CR feedback

043d766

Update config

fdc701f

Leave 3.1 for now per Dongjoon saying for K8s GA

978e513

Style fixes

e17e558

Back to 3.2 per Dongjoon conversation

8c05e6e

Fixes

72a1647

Mark API as unstable & other CR feedback

ab0049c

Fix up imports

df4e4b7

Remove unused import

e990f33

holdenk force-pushed the SPARK-33261-allow-people-to-extend-pod-feature-steps branch from 963b786 to e990f33 Compare December 11, 2020 22:49

Fix PodBuilderSuite to not use the private class

cfe5bc1

asfgit closed this in 5885cc1 Dec 14, 2020

Yikun mentioned this pull request Dec 16, 2021

[SPARK-37145][K8S] Add support for extending user feature steps with configuration #34924

Closed

Yikun mentioned this pull request Jan 28, 2022

[SPARK-37145][K8S] Add KubernetesCustom[Driver/Executor]FeatureConfigStep developer api #35345

Closed

sbernauer mentioned this pull request Mar 18, 2022

Make dependencies - spark and otherwise - available to spark pods externally via e.g. volumes, s3 etc. stackabletech/spark-k8s-operator#6

Closed

dongjoon-hyun mentioned this pull request Jul 16, 2025

[SPARK-52830][K8S] Support spark.kubernetes.(driver|executor).pod.excludedFeatureSteps #51522

Closed

[SPARK-33261][K8S] Add a developer API for custom feature steps #30206

[SPARK-33261][K8S] Add a developer API for custom feature steps #30206

Uh oh!

Conversation

holdenk commented Oct 30, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Oct 30, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 30, 2020

Uh oh!

SparkQA commented Oct 30, 2020

Uh oh!

holdenk commented Oct 30, 2020

Uh oh!

SparkQA commented Oct 30, 2020

Uh oh!

SparkQA commented Oct 30, 2020

Uh oh!

SparkQA commented Oct 30, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tgravescs commented Nov 2, 2020

Uh oh!

holdenk commented Nov 3, 2020

Uh oh!

holdenk commented Dec 3, 2020

Uh oh!

tgravescs commented Dec 3, 2020

Uh oh!

dongjoon-hyun commented Dec 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Dec 9, 2020

Uh oh!

tgravescs commented Dec 9, 2020

Uh oh!

dongjoon-hyun commented Dec 9, 2020

Uh oh!

dongjoon-hyun commented Dec 9, 2020

Uh oh!

SparkQA commented Dec 9, 2020

Uh oh!

Uh oh!

SparkQA commented Dec 11, 2020

Uh oh!

SparkQA commented Dec 11, 2020

Uh oh!

SparkQA commented Dec 12, 2020

Uh oh!

SparkQA commented Dec 12, 2020

Uh oh!

SparkQA commented Dec 12, 2020

Uh oh!

SparkQA commented Dec 12, 2020

Uh oh!

holdenk commented Dec 14, 2020

Uh oh!

Uh oh!

dongjoon-hyun commented Dec 6, 2020 •

edited

Loading