Skip to content

Commit

Permalink
Only parse the properties file if special configs exist
Browse files Browse the repository at this point in the history
This is to minimize the scope of influence exerted by this PR.
In the future, we should make an effort to route all usages of
spark-class through SparkClassLauncher so as to avoid duplicate
logic in bash and Scala.
  • Loading branch information
andrewor14 committed Aug 19, 2014
1 parent c37e08d commit 3a8235d
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 5 deletions.
1 change: 1 addition & 0 deletions bin/spark-class
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@ fi
# the driver JVM itself.

if [ -n "$SPARK_SUBMIT_CLIENT_MODE" ]; then
# This is currently used only if the properties file actually consists of these special configs
exec "$RUNNER" org.apache.spark.deploy.SparkClassLauncher \
"$PROPERTIES_FILE" \
"$RUNNER" \
Expand Down
17 changes: 12 additions & 5 deletions bin/spark-submit
Original file line number Diff line number Diff line change
Expand Up @@ -42,16 +42,23 @@ DEFAULT_PROPERTIES_FILE="$SPARK_HOME/conf/spark-defaults.conf"
PROPERTIES_FILE=${PROPERTIES_FILE:-"$DEFAULT_PROPERTIES_FILE"}

# For client mode, the driver will be launched in the same JVM that launches
# SparkSubmit, so we need to read the properties file for any extra class paths,
# library paths, java options and memory early on. Otherwise, it will be too
# late by the time the JVM has started.
# SparkSubmit, so we may need to read the properties file for any extra class
# paths, library paths, java options and memory early on. Otherwise, it will
# be too late by the time the JVM has started.

if [ "$DEPLOY_MODE" == "client" ]; then
if [ -n "$DRIVER_MEMORY" ]; then
export SPARK_DRIVER_MEMORY=$DRIVER_MEMORY
fi
export PROPERTIES_FILE
export SPARK_SUBMIT_CLIENT_MODE=1
# Parse the properties file only if the special configs exist
contains_special_configs=$(
grep -e "spark.driver.extra*\|spark.driver.memory" "$PROPERTIES_FILE" | \
grep -v "^[[:space:]]*#"
)
if [ -n "$contains_special_configs" ]; then
export PROPERTIES_FILE
export SPARK_SUBMIT_CLIENT_MODE=1
fi
fi

exec $SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit "${ORIG_ARGS[@]}"
Expand Down

0 comments on commit 3a8235d

Please sign in to comment.