Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SW-368] pysparkling Windows shell script #225

Merged
merged 1 commit into from Mar 27, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 6 additions & 0 deletions bin/pysparkling.cmd
@@ -0,0 +1,6 @@
@echo off

rem This is the entry point for running Spark shell. To avoid polluting the
rem environment, it just launches a new cmd to do the real work.

cmd /V /E /C %~dp0pysparkling2.cmd %*
23 changes: 23 additions & 0 deletions bin/pysparkling2.cmd
@@ -0,0 +1,23 @@
@echo off

rem Top-level directory for this product
set TOPDIR=%~dp0..
call %TOPDIR%\bin\sparkling-env.cmd
rem Verify there is Spark installation
call %LIBSW% checkSparkHome
call %LIBSW% checkSparkVersion
rem end of checking Sparkling environment

call %LIBSW% :banner

rem setup pysparkling command line
rem Because of SPARK-18648 we need to put assembly also on driver/executor class paths
SET SPARK_OPT_JARS=--jars !FAT_JAR_FILE! --conf spark.driver.extraClassPath=!FAT_JAR_FILE! --conf spark.executor.extraClassPath=!FAT_JAR_FILE!
SET SPARK_OPT_PYFILES=--py-files !PY_EGG_FILE!
cd %TOPDIR%
SET PYTHONPATH=%PY_EGG_FILE%:%PYTHONPATH%
call !SPARK_HOME!/bin/pyspark2.cmd !SPARK_OPT_PYFILES! %SPARK_OPT_JARS% %*

exit /b %ERRORLEVEL%
rem end of main script

4 changes: 4 additions & 0 deletions bin/sparkling-env.cmd
Expand Up @@ -15,10 +15,14 @@ for /f "tokens=2 delims==" %%i in ('TYPE %TOPDIR%\gradle.properties ^| findstr /

rem Ensure that scala version contains only major version
for /f "tokens=1,2 delims=." %%j in ("%SCALA_PARSED_VERSION%") do (@set SCALA_VERSION=%%j.%%k)
for /f "tokens=1,2 delims=." %%i in ("%SPARK_VERSION%") do (@set SPARK_MAJOR_VERSION=%%i.%%j)
for /f "tokens=1 delims=-" %%i in ("%VERSION%") do (@set VERSION_WITHOUT_SNAPSHOT=%%i)

rem Fat jar for this distribution
set FAT_JAR=sparkling-water-assembly_%SCALA_VERSION%-%VERSION%-all.jar
set FAT_JAR_FILE=%TOPDIR%\assembly\build\libs\%FAT_JAR%
set PY_EGG=h2o_pysparkling_%SPARK_MAJOR_VERSION%-%VERSION_WITHOUT_SNAPSHOT%-py2.7.egg
set PY_EGG_FILE=%TOPDIR%\py\build\dist\%PY_EGG%

rem Setup loging and output

Expand Down