Skip to content

Commit

Permalink
MAPR-17422: Ensure only Spark log4j.properties is picked up.
Browse files Browse the repository at this point in the history
When SPARK_CLASSPATH or SPARK_SUBMIT_CLASSPATH contains log4j.properties, they
will get picked up instead of Spark's version because they are present before
the Spark conf dir in classpath.

Testing:
Verified the INFO Messages don't get printed when the level is set to WARN.
  • Loading branch information
rkannan82 committed Apr 22, 2015
1 parent f1d0a78 commit 89d273a
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 4 deletions.
5 changes: 4 additions & 1 deletion bin/compute-classpath.cmd
Expand Up @@ -36,14 +36,17 @@ rem Load environment variables from conf\spark-env.cmd, if it exists
if exist "%FWDIR%conf\spark-env.cmd" call "%FWDIR%conf\spark-env.cmd"

rem Build up classpath
set CLASSPATH=%SPARK_CLASSPATH%;%SPARK_SUBMIT_CLASSPATH%

if not "x%SPARK_CONF_DIR%"=="x" (
set CLASSPATH=%CLASSPATH%;%SPARK_CONF_DIR%
) else (
set CLASSPATH=%CLASSPATH%;%FWDIR%conf
)

rem Add the external jars after adding spark conf so that Spark's log4j.properties
rem is picked up.
set CLASSPATH=%CLASSPATH%;%SPARK_CLASSPATH%;%SPARK_SUBMIT_CLASSPATH%

if exist "%FWDIR%RELEASE" (
for %%d in ("%FWDIR%lib\spark-assembly*.jar") do (
set ASSEMBLY_JAR=%%d
Expand Down
8 changes: 5 additions & 3 deletions bin/compute-classpath.sh
Expand Up @@ -37,16 +37,18 @@ function appendToClasspath(){
fi
}

appendToClasspath "$SPARK_CLASSPATH"
appendToClasspath "$SPARK_SUBMIT_CLASSPATH"

# Build up classpath
if [ -n "$SPARK_CONF_DIR" ]; then
appendToClasspath "$SPARK_CONF_DIR"
else
appendToClasspath "$FWDIR/conf"
fi

# Add the external jars after adding spark conf so that Spark's log4j.properties
appendToClasspath "$SPARK_CLASSPATH"
appendToClasspath "$SPARK_SUBMIT_CLASSPATH"


ASSEMBLY_DIR="$FWDIR/assembly/target/scala-$SPARK_SCALA_VERSION"

if [ -n "$JAVA_HOME" ]; then
Expand Down

0 comments on commit 89d273a

Please sign in to comment.