Skip to content

Commit

Permalink
[SPARK-18645][DEPLOY] Fix spark-daemon.sh arguments error lead to thr…
Browse files Browse the repository at this point in the history
…ows Unrecognized option

## What changes were proposed in this pull request?

spark-daemon.sh will lost single quotes around after #15338. as follows:
```
execute_command nice -n 0 bash /opt/cloudera/parcels/SPARK-2.1.0-cdh5.4.3.d20161129-21.04.38/lib/spark/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server --conf spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp
```
With this fix, as follows:
```
execute_command nice -n 0 bash /opt/cloudera/parcels/SPARK-2.1.0-cdh5.4.3.d20161129-21.04.38/lib/spark/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name 'Thrift JDBC/ODBC Server' --conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp'
```

## How was this patch tested?

- Manual tests
- Build the package and start-thriftserver.sh with `--conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp'`

Author: Yuming Wang <wgyumg@gmail.com>

Closes #16079 from wangyum/SPARK-18645.

(cherry picked from commit 2ab8551)
Signed-off-by: Sean Owen <sowen@cloudera.com>
  • Loading branch information
wangyum authored and srowen committed Dec 1, 2016
1 parent 8579ab5 commit cbbe217
Showing 1 changed file with 5 additions and 6 deletions.
11 changes: 5 additions & 6 deletions sbin/spark-daemon.sh
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,8 @@ if [ "$SPARK_NICENESS" = "" ]; then
fi

execute_command() {
local command="$@"
if [ -z ${SPARK_NO_DAEMONIZE+set} ]; then
nohup -- $command >> $log 2>&1 < /dev/null &
nohup -- "$@" >> $log 2>&1 < /dev/null &
newpid="$!"

echo "$newpid" > "$pid"
Expand All @@ -143,12 +142,12 @@ execute_command() {
sleep 2
# Check if the process has died; in that case we'll tail the log so the user can see
if [[ ! $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
echo "failed to launch $command:"
echo "failed to launch: $@"
tail -2 "$log" | sed 's/^/ /'
echo "full log in $log"
fi
else
$command
"$@"
fi
}

Expand Down Expand Up @@ -176,11 +175,11 @@ run_command() {

case "$mode" in
(class)
execute_command nice -n "$SPARK_NICENESS" "${SPARK_HOME}"/bin/spark-class $command $@
execute_command nice -n "$SPARK_NICENESS" "${SPARK_HOME}"/bin/spark-class "$command" "$@"
;;

(submit)
execute_command nice -n "$SPARK_NICENESS" bash "${SPARK_HOME}"/bin/spark-submit --class $command $@
execute_command nice -n "$SPARK_NICENESS" bash "${SPARK_HOME}"/bin/spark-submit --class "$command" "$@"
;;

(*)
Expand Down

0 comments on commit cbbe217

Please sign in to comment.