Skip to content

Commit

Permalink
[SPARK-30970][K8S][CORE][2.4] Fix NPE while resolving k8s master url
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

backport #27721
```
bin/spark-sql --master  k8s:///https://kubernetes.docker.internal:6443 --conf spark.kubernetes.container.image=yaooqinn/spark:v2.4.4
Exception in thread "main" java.lang.NullPointerException
	at org.apache.spark.util.Utils$.checkAndGetK8sMasterUrl(Utils.scala:2739)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:261)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
Althrough `k8s:///https://kubernetes.docker.internal:6443` is a wrong master url but should not throw npe
The `case null` will never be touched.
https://github.com/apache/spark/blob/3f4060c340d6bac412e8819c4388ccba226efcf3/core/src/main/scala/org/apache/spark/util/Utils.scala#L2772-L2776

### Why are the changes needed?

bug fix

### Does this PR introduce any user-facing change?

no

### How was this patch tested?

add ut case

Closes #27736 from yaooqinn/SPARK-30970-2.

Authored-by: Kent Yao <yaooqinn@hotmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
  • Loading branch information
yaooqinn authored and dongjoon-hyun committed Feb 28, 2020
1 parent 7574d99 commit ff5ba49
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 9 deletions.
15 changes: 6 additions & 9 deletions core/src/main/scala/org/apache/spark/util/Utils.scala
Original file line number Diff line number Diff line change
Expand Up @@ -2736,19 +2736,16 @@ private[spark] object Utils extends Logging {
}

val masterScheme = new URI(masterWithoutK8sPrefix).getScheme
val resolvedURL = masterScheme.toLowerCase match {
case "https" =>

val resolvedURL = Option(masterScheme).map(_.toLowerCase(Locale.ROOT)) match {
case Some("https") =>
masterWithoutK8sPrefix
case "http" =>
case Some("http") =>
logWarning("Kubernetes master URL uses HTTP instead of HTTPS.")
masterWithoutK8sPrefix
case null =>
val resolvedURL = s"https://$masterWithoutK8sPrefix"
logInfo("No scheme specified for kubernetes master URL, so defaulting to https. Resolved " +
s"URL is $resolvedURL.")
resolvedURL
case _ =>
throw new IllegalArgumentException("Invalid Kubernetes master scheme: " + masterScheme)
throw new IllegalArgumentException("Invalid Kubernetes master scheme: " + masterScheme
+ " found in URL: " + masterWithoutK8sPrefix)
}

s"k8s://$resolvedURL"
Expand Down
4 changes: 4 additions & 0 deletions core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
Original file line number Diff line number Diff line change
Expand Up @@ -1167,6 +1167,10 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties with Logging {
intercept[IllegalArgumentException] {
Utils.checkAndGetK8sMasterUrl("k8s://foo://host:port")
}

intercept[IllegalArgumentException] {
Utils.checkAndGetK8sMasterUrl("k8s:///https://host:port")
}
}

object MalformedClassObject {
Expand Down

0 comments on commit ff5ba49

Please sign in to comment.