[SPARK-30970][K8S][Core] Fix NPE while resolving k8s master url #27721

yaooqinn · 2020-02-27T09:43:14Z

What changes were proposed in this pull request?

bin/spark-sql --master  k8s:///https://kubernetes.docker.internal:6443 --conf spark.kubernetes.container.image=yaooqinn/spark:v2.4.4
Exception in thread "main" java.lang.NullPointerException
	at org.apache.spark.util.Utils$.checkAndGetK8sMasterUrl(Utils.scala:2739)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:261)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Althrough k8s:///https://kubernetes.docker.internal:6443 is a wrong master url but should not throw npe
The case null will never be touched.

spark/core/src/main/scala/org/apache/spark/util/Utils.scala

Lines 2772 to 2776 in 3f4060c

    
           case null => 
        
             val resolvedURL = s"https://$masterWithoutK8sPrefix" 
        
             logInfo("No scheme specified for kubernetes master URL, so defaulting to https. Resolved " + 
        
               s"URL is $resolvedURL.") 
        
             resolvedURL

Why are the changes needed?

bug fix

Does this PR introduce any user-facing change?

no

How was this patch tested?

add ut case

yaooqinn · 2020-02-27T09:44:22Z

cc @dongjoon-hyun @srowen, thanks for reviewing

core/src/main/scala/org/apache/spark/util/Utils.scala

SparkQA · 2020-02-27T10:42:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23770/

SparkQA · 2020-02-27T11:05:42Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23770/

SparkQA · 2020-02-27T13:03:07Z

Test build #119022 has finished for PR 27721 at commit 6a7d390.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-27T13:30:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23776/

SparkQA · 2020-02-27T13:55:07Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23776/

yaooqinn · 2020-02-27T16:07:39Z

retest this please

SparkQA · 2020-02-27T16:18:10Z

Test build #119028 has finished for PR 27721 at commit b3dcb63.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-27T16:55:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23792/

srowen

OK I buy it.

SparkQA · 2020-02-27T17:25:33Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23792/

SparkQA · 2020-02-27T18:51:46Z

Test build #119046 has finished for PR 27721 at commit b3dcb63.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2020-02-28T00:43:34Z

retest this please

SparkQA · 2020-02-28T01:23:34Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23806/

SparkQA · 2020-02-28T01:59:57Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23806/

SparkQA · 2020-02-28T03:56:16Z

Test build #119062 has finished for PR 27721 at commit b3dcb63.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2020-02-28T04:36:37Z

retest this please

SparkQA · 2020-02-28T05:55:56Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23812/

SparkQA · 2020-02-28T06:23:04Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23812/

yaooqinn · 2020-02-28T06:25:04Z

retest this please, the k8s it test seems quite flaky.

SparkQA · 2020-02-28T07:03:18Z

Test build #119067 has finished for PR 27721 at commit b3dcb63.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-02-28T07:10:44Z

cc @holdenk for the flaky test which is blocking this PR.

- Test basic decommissioning *** FAILED ***
  The code passed to eventually never returned normally. Attempted 126 times over 2.0100952450666667 minutes. Last failure message: "++ id -u

I created SPARK-30981 for the flaky test.

SparkQA · 2020-02-28T07:34:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23815/

SparkQA · 2020-02-28T07:56:19Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23815/

dongjoon-hyun

+1, LGTM. Now all tests passed.

[SPARK-30970][K8S][Core] Fix NPE while resolving k8s master url #27721 (comment)
[SPARK-30970][K8S][Core] Fix NPE while resolving k8s master url #27721 (comment)

Merged to master/3.0.

### What changes were proposed in this pull request? ``` bin/spark-sql --master k8s:///https://kubernetes.docker.internal:6443 --conf spark.kubernetes.container.image=yaooqinn/spark:v2.4.4 Exception in thread "main" java.lang.NullPointerException at org.apache.spark.util.Utils$.checkAndGetK8sMasterUrl(Utils.scala:2739) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:261) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ``` Althrough `k8s:///https://kubernetes.docker.internal:6443` is a wrong master url but should not throw npe The `case null` will never be touched. https://github.com/apache/spark/blob/3f4060c340d6bac412e8819c4388ccba226efcf3/core/src/main/scala/org/apache/spark/util/Utils.scala#L2772-L2776 ### Why are the changes needed? bug fix ### Does this PR introduce any user-facing change? no ### How was this patch tested? add ut case Closes #27721 from yaooqinn/SPARK-30970. Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 1383bd4) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

dongjoon-hyun · 2020-02-28T08:03:23Z

Hi, @yaooqinn . Could you make a PR to branch-2.4? There is a conflict now.

SparkQA · 2020-02-28T08:05:01Z

Test build #119071 has finished for PR 27721 at commit b3dcb63.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2020-02-28T08:43:30Z

OK @dongjoon-hyun and thanks for merging

### What changes were proposed in this pull request? backport #27721 ``` bin/spark-sql --master k8s:///https://kubernetes.docker.internal:6443 --conf spark.kubernetes.container.image=yaooqinn/spark:v2.4.4 Exception in thread "main" java.lang.NullPointerException at org.apache.spark.util.Utils$.checkAndGetK8sMasterUrl(Utils.scala:2739) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:261) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ``` Althrough `k8s:///https://kubernetes.docker.internal:6443` is a wrong master url but should not throw npe The `case null` will never be touched. https://github.com/apache/spark/blob/3f4060c340d6bac412e8819c4388ccba226efcf3/core/src/main/scala/org/apache/spark/util/Utils.scala#L2772-L2776 ### Why are the changes needed? bug fix ### Does this PR introduce any user-facing change? no ### How was this patch tested? add ut case Closes #27736 from yaooqinn/SPARK-30970-2. Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? ``` bin/spark-sql --master k8s:///https://kubernetes.docker.internal:6443 --conf spark.kubernetes.container.image=yaooqinn/spark:v2.4.4 Exception in thread "main" java.lang.NullPointerException at org.apache.spark.util.Utils$.checkAndGetK8sMasterUrl(Utils.scala:2739) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:261) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ``` Althrough `k8s:///https://kubernetes.docker.internal:6443` is a wrong master url but should not throw npe The `case null` will never be touched. https://github.com/apache/spark/blob/3f4060c340d6bac412e8819c4388ccba226efcf3/core/src/main/scala/org/apache/spark/util/Utils.scala#L2772-L2776 ### Why are the changes needed? bug fix ### Does this PR introduce any user-facing change? no ### How was this patch tested? add ut case Closes apache#27721 from yaooqinn/SPARK-30970. Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

[SPARK-30970][K8S][Core] Fix NPE while resolving k8s master url

6a7d390

dongjoon-hyun added the KUBERNETES label Feb 27, 2020

dongjoon-hyun reviewed Feb 27, 2020

View reviewed changes

core/src/main/scala/org/apache/spark/util/Utils.scala Show resolved Hide resolved

improve err msg

b3dcb63

srowen approved these changes Feb 27, 2020

View reviewed changes

dongjoon-hyun added the SPARK CORE label Feb 28, 2020

dongjoon-hyun approved these changes Feb 28, 2020

View reviewed changes

dongjoon-hyun closed this in 1383bd4 Feb 28, 2020

yaooqinn mentioned this pull request Feb 28, 2020

[SPARK-30970][K8S][Core][2.4] Fix NPE while resolving k8s master url #27736

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-30970][K8S][Core] Fix NPE while resolving k8s master url #27721

[SPARK-30970][K8S][Core] Fix NPE while resolving k8s master url #27721

yaooqinn commented Feb 27, 2020

yaooqinn commented Feb 27, 2020

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

yaooqinn commented Feb 27, 2020

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

srowen left a comment

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

yaooqinn commented Feb 28, 2020

SparkQA commented Feb 28, 2020

SparkQA commented Feb 28, 2020

SparkQA commented Feb 28, 2020

yaooqinn commented Feb 28, 2020

SparkQA commented Feb 28, 2020

SparkQA commented Feb 28, 2020

yaooqinn commented Feb 28, 2020

SparkQA commented Feb 28, 2020

dongjoon-hyun commented Feb 28, 2020 •

edited

Loading

SparkQA commented Feb 28, 2020

SparkQA commented Feb 28, 2020

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun commented Feb 28, 2020

SparkQA commented Feb 28, 2020

yaooqinn commented Feb 28, 2020

	case null =>
	val resolvedURL = s"https://$masterWithoutK8sPrefix"
	logInfo("No scheme specified for kubernetes master URL, so defaulting to https. Resolved " +
	s"URL is $resolvedURL.")
	resolvedURL

[SPARK-30970][K8S][Core] Fix NPE while resolving k8s master url #27721

[SPARK-30970][K8S][Core] Fix NPE while resolving k8s master url #27721

Conversation

yaooqinn commented Feb 27, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

yaooqinn commented Feb 27, 2020

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

yaooqinn commented Feb 27, 2020

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

srowen left a comment

Choose a reason for hiding this comment

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

yaooqinn commented Feb 28, 2020

SparkQA commented Feb 28, 2020

SparkQA commented Feb 28, 2020

SparkQA commented Feb 28, 2020

yaooqinn commented Feb 28, 2020

SparkQA commented Feb 28, 2020

SparkQA commented Feb 28, 2020

yaooqinn commented Feb 28, 2020

SparkQA commented Feb 28, 2020

dongjoon-hyun commented Feb 28, 2020 • edited Loading

SparkQA commented Feb 28, 2020

SparkQA commented Feb 28, 2020

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun commented Feb 28, 2020

SparkQA commented Feb 28, 2020

yaooqinn commented Feb 28, 2020

dongjoon-hyun commented Feb 28, 2020 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading