[SPARK-33720][K8S] Support submit to k8s only with token #30684

hddong · 2020-12-09T10:08:56Z

What changes were proposed in this pull request?

Support submit to k8s only with token.

Why are the changes needed?

Now, sumbit to k8s always need oauth files.

Does this PR introduce any user-facing change?

How was this patch tested?

Before, submit job out of k8s cluster without correct ca.crt, we may get this exception:

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at sun.security.validator.Validator.validate(Validator.java:271)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:312)

When set spark.kubernetes.trust.certificates = true, we can submit only with correct token, no need to config ca.crt in local env.
Submit as:

 bin/spark-submit \
     --master $master \
     --name pi \
     --deploy-mode cluster \
     --conf spark.kubernetes.container.image=$image \
     --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
     --conf spark.kubernetes.authenticate.submission.oauthToken=$clusterToken \
     --conf spark.kubernetes.trust.certificates=true \
     local:///opt/spark/examples/src/main/python/pi.py 200

dongjoon-hyun · 2020-12-09T17:25:45Z

ok to test

dongjoon-hyun · 2020-12-09T17:26:39Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

+  val KUBERNETES_TRUST_CERTIFICATES =
+    ConfigBuilder("spark.kubernetes.trust.certificates")
+      .doc("If set to true then client can submit to kubernetes cluster only with token")
+      .version("3.0.2")


Hi, @hddong .
This should be 3.2.0 because this is a new feature and master branch's version is 3.2.0-SNAPSHOT.

dongjoon-hyun

In addition to https://github.com/apache/spark/pull/30684/files#r539500957, could you describe the environment where this PR is helpful? How can we verify this PR? Without testing or verification, this PR is not mergeable. Please write the reproducible procedure in the PR description instead of saying no need, please.

How was this patch tested?

no need

SparkQA · 2020-12-09T17:56:49Z

Test build #132501 has finished for PR 30684 at commit 8ddb290.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-09T18:22:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37103/

SparkQA · 2020-12-09T18:53:59Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37103/

hddong · 2020-12-14T09:56:42Z

@dongjoon-hyun : thansk for your review, had address them.

SparkQA · 2020-12-14T10:15:25Z

Test build #132767 has finished for PR 30684 at commit 14e1cac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-14T10:53:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37369/

SparkQA · 2020-12-14T11:21:01Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37369/

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

dongjoon-hyun

@hddong Why don't we use insecure-skip-tls-verify in .kubeconfig?

AmplabJenkins · 2021-01-03T17:00:35Z

Can one of the admins verify this patch?

hddong · 2021-01-19T10:02:15Z

@dongjoon-hyun : sorry for feedback so long time later, insecure-skip-tls-verify is a chose when have .kubeconfig. Use oauthToken, we can submit without .kubeconfig, and submit easy with master url and oauthToken.

dongjoon-hyun · 2021-01-19T19:25:55Z

I'm wondering when .kube/config is missing. In general, Apache Spark assumes an environment like A running Kubernetes cluster at version >= 1.6 with access configured to it using kubectl.

sorry for feedback so long time later, insecure-skip-tls-verify is a chose when have .kubeconfig. Use oauthToken, we can submit without .kubeconfig, and submit easy with master url and oauthToken.

hddong · 2021-01-20T09:10:31Z

@dongjoon-hyun : We can submit directly with token when client has no k8s client(kubectl and .kube/config) in environment. Otherwise, when submit to a new k8s cluster, we need modify .kube/config each time. kubernetes.trust.certificates make it simpler to submit our jars.

jho · 2021-02-22T20:51:41Z

I have a project that would greatly benefit from this patch as well. The ability to use .kube/config is nice but being able to use the spark-submit params to adjust what cluster we are submitting to dynamically is a better approach for us as well. AWS EKS k8s clusters use self-signed load balanced certs per cluster and this leaves with lots of issues with PKIX errors.

hddong · 2021-03-24T02:27:28Z

@dongjoon-hyun: how do you think about this PR.

dongjoon-hyun · 2021-03-24T04:24:42Z

Sorry for the delay, @hddong . I'll review right now again.

dongjoon-hyun

+1, LGTM. I verified this manually.
Merged to master for Apache Spark 3.2.0.

dongjoon-hyun · 2021-03-24T05:08:04Z

cc @attilapiros , too.

### What changes were proposed in this pull request? Support submit to k8s only with token. ### Why are the changes needed? Now, sumbit to k8s always need oauth files. ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Before, submit job out of k8s cluster without correct ca.crt, we may get this exception: ``` Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at sun.security.validator.Validator.validate(Validator.java:271) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:312) ``` When set spark.kubernetes.trust.certificates = true, we can submit only with correct token, no need to config ca.crt in local env. Submit as: ``` bin/spark-submit \ --master $master \ --name pi \ --deploy-mode cluster \ --conf spark.kubernetes.container.image=$image \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.kubernetes.authenticate.submission.oauthToken=$clusterToken \ --conf spark.kubernetes.trust.certificates=true \ local:///opt/spark/examples/src/main/python/pi.py 200 ``` Closes apache#30684 from hddong/trust-certs. Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 985c653) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

[SPARK-33720][Kubernetes]Support submit to k8s only with token

8ddb290

github-actions bot added the KUBERNETES label Dec 9, 2020

dongjoon-hyun changed the title ~~[SPARK-33720][Kubernetes]Support submit to k8s only with token~~ [SPARK-33720][K8S] Support submit to k8s only with token Dec 9, 2020

dongjoon-hyun reviewed Dec 9, 2020

View reviewed changes

dongjoon-hyun requested changes Dec 9, 2020

View reviewed changes

fix version

14e1cac

dongjoon-hyun reviewed Dec 31, 2020

View reviewed changes

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Show resolved Hide resolved

dongjoon-hyun reviewed Dec 31, 2020

View reviewed changes

dongjoon-hyun approved these changes Mar 24, 2021

View reviewed changes

dongjoon-hyun closed this in 985c653 Mar 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-33720][K8S] Support submit to k8s only with token #30684

[SPARK-33720][K8S] Support submit to k8s only with token #30684

hddong commented Dec 9, 2020 •

edited by dongjoon-hyun

dongjoon-hyun commented Dec 9, 2020

dongjoon-hyun Dec 9, 2020

dongjoon-hyun left a comment •

edited

SparkQA commented Dec 9, 2020

SparkQA commented Dec 9, 2020

SparkQA commented Dec 9, 2020

hddong commented Dec 14, 2020

SparkQA commented Dec 14, 2020

SparkQA commented Dec 14, 2020

SparkQA commented Dec 14, 2020

dongjoon-hyun left a comment

AmplabJenkins commented Jan 3, 2021

hddong commented Jan 19, 2021 •

edited

dongjoon-hyun commented Jan 19, 2021

hddong commented Jan 20, 2021

jho commented Feb 22, 2021

hddong commented Mar 24, 2021

dongjoon-hyun commented Mar 24, 2021 •

edited

dongjoon-hyun left a comment

dongjoon-hyun commented Mar 24, 2021

[SPARK-33720][K8S] Support submit to k8s only with token #30684

[SPARK-33720][K8S] Support submit to k8s only with token #30684

Conversation

hddong commented Dec 9, 2020 • edited by dongjoon-hyun

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented Dec 9, 2020

dongjoon-hyun Dec 9, 2020

Choose a reason for hiding this comment

dongjoon-hyun left a comment • edited

Choose a reason for hiding this comment

How was this patch tested?

SparkQA commented Dec 9, 2020

SparkQA commented Dec 9, 2020

SparkQA commented Dec 9, 2020

hddong commented Dec 14, 2020

SparkQA commented Dec 14, 2020

SparkQA commented Dec 14, 2020

SparkQA commented Dec 14, 2020

dongjoon-hyun left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Jan 3, 2021

hddong commented Jan 19, 2021 • edited

dongjoon-hyun commented Jan 19, 2021

hddong commented Jan 20, 2021

jho commented Feb 22, 2021

hddong commented Mar 24, 2021

dongjoon-hyun commented Mar 24, 2021 • edited

dongjoon-hyun left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented Mar 24, 2021

hddong commented Dec 9, 2020 •

edited by dongjoon-hyun

dongjoon-hyun left a comment •

edited

hddong commented Jan 19, 2021 •

edited

dongjoon-hyun commented Mar 24, 2021 •

edited