[SPARK-31601][K8S] Fix spark.kubernetes.executor.podNamePrefix to work #28401

dongjoon-hyun · 2020-04-29T06:25:33Z

What changes were proposed in this pull request?

This PR aims to fix spark.kubernetes.executor.podNamePrefix to work.

Why are the changes needed?

Currently, the configuration is broken like the following.

bin/spark-submit \
--master k8s://$K8S_MASTER \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
-c spark.kubernetes.container.image=spark:pr \
-c spark.kubernetes.driver.pod.name=mypod \
-c spark.kubernetes.executor.podNamePrefix=mypod \
local:///opt/spark/examples/jars/spark-examples_2.12-3.1.0-SNAPSHOT.jar

BEFORE SPARK-31601

pod/mypod                              1/1     Running     0          9s
pod/spark-pi-7469dd71c499fafb-exec-1   1/1     Running     0          4s
pod/spark-pi-7469dd71c499fafb-exec-2   1/1     Running     0          4s

AFTER SPARK-31601

pod/mypod                              1/1     Running     0          8s
pod/mypod-exec-1                       1/1     Running     0          3s
pod/mypod-exec-2                       1/1     Running     0          3s

Does this PR introduce any user-facing change?

Yes. This is a bug fix. The conf will work as described in the documentation.

How was this patch tested?

Pass the Jenkins and run the above comment manually.

...rnetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala

ScrapCodes · 2020-04-29T07:58:16Z

Hi @dongjoon-hyun , thanks for the fix !

Run SparkPi with no resources *** FAILED ***
The code passed to eventually never returned normally. Attempted 130 times over 2.0021201478333333 minutes. Last failure message: false was not true. (KubernetesSuite.scala:370)

^^^ Seems to be a flaky test.

ScrapCodes · 2020-04-29T07:58:25Z

retest

...rce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala

...rnetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala

dongjoon-hyun · 2020-04-29T16:00:35Z

Thank you so much for review, @ScrapCodes .

SparkQA · 2020-04-29T16:14:37Z

Test build #122073 has finished for PR 28401 at commit c1e181d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-04-29T18:02:24Z

Retest this please.

SparkQA · 2020-04-29T18:23:19Z

Test build #122083 has finished for PR 28401 at commit c1e181d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-29T18:56:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26758/

dongjoon-hyun · 2020-04-29T19:01:58Z

cc @holdenk

SparkQA · 2020-04-29T19:14:13Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26758/

dongjoon-hyun · 2020-04-30T03:00:57Z

Could you review this once more, @ScrapCodes ?

ScrapCodes · 2020-04-30T03:27:25Z

Thanks ! @dongjoon-hyun , LGTM :)

ScrapCodes · 2020-04-30T03:29:51Z

I wish we had a merge bot, which could detect a LGTM from a committer and do the need full.

ScrapCodes · 2020-04-30T03:42:31Z

I am going to merge this into Master and branch-3.0.

### What changes were proposed in this pull request? This PR aims to fix `spark.kubernetes.executor.podNamePrefix` to work. ### Why are the changes needed? Currently, the configuration is broken like the following. ``` bin/spark-submit \ --master k8s://$K8S_MASTER \ --deploy-mode cluster \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ -c spark.kubernetes.container.image=spark:pr \ -c spark.kubernetes.driver.pod.name=mypod \ -c spark.kubernetes.executor.podNamePrefix=mypod \ local:///opt/spark/examples/jars/spark-examples_2.12-3.1.0-SNAPSHOT.jar ``` **BEFORE SPARK-31601** ``` pod/mypod 1/1 Running 0 9s pod/spark-pi-7469dd71c499fafb-exec-1 1/1 Running 0 4s pod/spark-pi-7469dd71c499fafb-exec-2 1/1 Running 0 4s ``` **AFTER SPARK-31601** ``` pod/mypod 1/1 Running 0 8s pod/mypod-exec-1 1/1 Running 0 3s pod/mypod-exec-2 1/1 Running 0 3s ``` ### Does this PR introduce any user-facing change? Yes. This is a bug fix. The conf will work as described in the documentation. ### How was this patch tested? Pass the Jenkins and run the above comment manually. Closes #28401 from dongjoon-hyun/SPARK-31601. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Prashant Sharma <prashsh1@in.ibm.com> (cherry picked from commit 85dad37) Signed-off-by: Prashant Sharma <prashsh1@in.ibm.com>

dongjoon-hyun · 2020-04-30T04:50:41Z

Thank you for review and merging, @ScrapCodes .
Ya. Bot is popular in these days. :)

This PR aims to fix `spark.kubernetes.executor.podNamePrefix` to work. Currently, the configuration is broken like the following. ``` bin/spark-submit \ --master k8s://$K8S_MASTER \ --deploy-mode cluster \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ -c spark.kubernetes.container.image=spark:pr \ -c spark.kubernetes.driver.pod.name=mypod \ -c spark.kubernetes.executor.podNamePrefix=mypod \ local:///opt/spark/examples/jars/spark-examples_2.12-3.1.0-SNAPSHOT.jar ``` **BEFORE SPARK-31601** ``` pod/mypod 1/1 Running 0 9s pod/spark-pi-7469dd71c499fafb-exec-1 1/1 Running 0 4s pod/spark-pi-7469dd71c499fafb-exec-2 1/1 Running 0 4s ``` **AFTER SPARK-31601** ``` pod/mypod 1/1 Running 0 8s pod/mypod-exec-1 1/1 Running 0 3s pod/mypod-exec-2 1/1 Running 0 3s ``` Yes. This is a bug fix. The conf will work as described in the documentation. Pass the Jenkins and run the above comment manually. Closes #28401 from dongjoon-hyun/SPARK-31601. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Prashant Sharma <prashsh1@in.ibm.com> (cherry picked from commit 85dad37) Signed-off-by: Prashant Sharma <prashsh1@in.ibm.com> (cherry picked from commit 82b8f7f) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2020-04-30T05:09:34Z

Since the bug exists in branch-2.4, I cherry-pick this to branch-2.4, too.

cc @holdenk

ScrapCodes · 2020-04-30T05:11:00Z

@dongjoon-hyun honestly, I was about to say that, thanks for doing it. :)

dongjoon-hyun · 2020-04-30T05:12:43Z

:)

[SPARK-31601][K8S] Fix spark.kubernetes.executor.podNamePrefix to work

e42d772

probot-autolabeler bot added the KUBERNETES label Apr 29, 2020

dongjoon-hyun commented Apr 29, 2020

View reviewed changes

...rnetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala Show resolved Hide resolved

break line

fd5378c

This comment has been minimized.

Sign in to view

dongjoon-hyun closed this Apr 29, 2020

dongjoon-hyun reopened this Apr 29, 2020

This comment has been minimized.

Sign in to view

ScrapCodes reviewed Apr 29, 2020

View reviewed changes

...rce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

ScrapCodes suggested changes Apr 29, 2020

View reviewed changes

...rnetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala Outdated Show resolved Hide resolved

Address comments

c1e181d

This comment has been minimized.

Sign in to view

ScrapCodes approved these changes Apr 30, 2020

View reviewed changes

asfgit closed this in 85dad37 Apr 30, 2020

dongjoon-hyun deleted the SPARK-31601 branch April 30, 2020 04:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-31601][K8S] Fix spark.kubernetes.executor.podNamePrefix to work #28401

[SPARK-31601][K8S] Fix spark.kubernetes.executor.podNamePrefix to work #28401

dongjoon-hyun commented Apr 29, 2020 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ScrapCodes commented Apr 29, 2020 •

edited

Loading

ScrapCodes commented Apr 29, 2020

This comment has been minimized.

dongjoon-hyun commented Apr 29, 2020

SparkQA commented Apr 29, 2020

This comment has been minimized.

This comment has been minimized.

dongjoon-hyun commented Apr 29, 2020

SparkQA commented Apr 29, 2020

SparkQA commented Apr 29, 2020

dongjoon-hyun commented Apr 29, 2020

SparkQA commented Apr 29, 2020

dongjoon-hyun commented Apr 30, 2020

ScrapCodes commented Apr 30, 2020

ScrapCodes commented Apr 30, 2020

ScrapCodes commented Apr 30, 2020

dongjoon-hyun commented Apr 30, 2020

dongjoon-hyun commented Apr 30, 2020

ScrapCodes commented Apr 30, 2020

dongjoon-hyun commented Apr 30, 2020

[SPARK-31601][K8S] Fix spark.kubernetes.executor.podNamePrefix to work #28401

[SPARK-31601][K8S] Fix spark.kubernetes.executor.podNamePrefix to work #28401

Conversation

dongjoon-hyun commented Apr 29, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ScrapCodes commented Apr 29, 2020 • edited Loading

ScrapCodes commented Apr 29, 2020

This comment has been minimized.

dongjoon-hyun commented Apr 29, 2020

SparkQA commented Apr 29, 2020

This comment has been minimized.

This comment has been minimized.

dongjoon-hyun commented Apr 29, 2020

SparkQA commented Apr 29, 2020

SparkQA commented Apr 29, 2020

dongjoon-hyun commented Apr 29, 2020

SparkQA commented Apr 29, 2020

dongjoon-hyun commented Apr 30, 2020

ScrapCodes commented Apr 30, 2020

ScrapCodes commented Apr 30, 2020

ScrapCodes commented Apr 30, 2020

dongjoon-hyun commented Apr 30, 2020

dongjoon-hyun commented Apr 30, 2020

ScrapCodes commented Apr 30, 2020

dongjoon-hyun commented Apr 30, 2020

dongjoon-hyun commented Apr 29, 2020 •

edited

Loading

ScrapCodes commented Apr 29, 2020 •

edited

Loading