-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-22962][K8S] Fail fast if submission client local files are used #20320
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -117,7 +117,10 @@ This URI is the location of the example jar that is already in the Docker image. | |
If your application's dependencies are all hosted in remote locations like HDFS or HTTP servers, they may be referred to | ||
by their appropriate remote URIs. Also, application dependencies can be pre-mounted into custom-built Docker images. | ||
Those dependencies can be added to the classpath by referencing them with `local://` URIs and/or setting the | ||
`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles. | ||
`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles. The `local://` scheme is also required when referring to | ||
dependencies in custom-built Docker images in `spark-submit`. Note that using application dependencies local to the submission | ||
client is currently not yet supported. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "application dependencies from the local file system"? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. |
||
|
||
|
||
### Using Remote Dependencies | ||
When there are application dependencies hosted in remote locations like HDFS or HTTP servers, the driver and executor pods | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,7 +20,7 @@ import java.util.UUID | |
|
||
import com.google.common.primitives.Longs | ||
|
||
import org.apache.spark.SparkConf | ||
import org.apache.spark.{SparkConf, SparkException} | ||
import org.apache.spark.deploy.k8s.{KubernetesUtils, MountSecretsBootstrap} | ||
import org.apache.spark.deploy.k8s.Config._ | ||
import org.apache.spark.deploy.k8s.Constants._ | ||
|
@@ -117,6 +117,12 @@ private[spark] class DriverConfigOrchestrator( | |
.map(_.split(",")) | ||
.getOrElse(Array.empty[String]) | ||
|
||
if (existSubmissionLocalFiles(sparkJars) || existSubmissionLocalFiles(sparkFiles)) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can add a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done, and also created https://issues.apache.org/jira/browse/SPARK-23153. |
||
throw new SparkException("The Kubernetes mode does not yet support application " + | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd shorten this to just "Kubernetes mode does not support referencing application dependencies in the local file system". There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. |
||
"dependencies local to the submission client. It currently only allows application" + | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: extra space in the end of line There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. |
||
"dependencies locally in the container or that can be downloaded remotely.") | ||
} | ||
|
||
val dependencyResolutionStep = if (sparkJars.nonEmpty || sparkFiles.nonEmpty) { | ||
Seq(new DependencyResolutionStep( | ||
sparkJars, | ||
|
@@ -162,6 +168,12 @@ private[spark] class DriverConfigOrchestrator( | |
initContainerBootstrapStep | ||
} | ||
|
||
private def existSubmissionLocalFiles(files: Seq[String]): Boolean = { | ||
files.exists { uri => | ||
Utils.resolveURI(uri).getScheme == "file" | ||
} | ||
} | ||
|
||
private def existNonContainerLocalFiles(files: Seq[String]): Boolean = { | ||
files.exists { uri => | ||
Utils.resolveURI(uri).getScheme != "local" | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The point above already covers that
local://
is needed with custom-built images.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but that's about adding to the classpath. I wanted to make it more specific and clearer.