Unable to start thunder with Spark 1.4.0

[naegelejd]

Following the installation instructions here on Github, I downloaded spark-1.4.0-bin-hadoop1 and set SPARK_HOME. I then installed thunder using pip. When I run thunder I get the following error:

(thunder)$ ~ thunder
Exception in thread "main" java.lang.IllegalArgumentException: pyspark does not support any application options.
    at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:222)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildPySparkShellCommand(SparkSubmitCommandBuilder.java:239)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:113)
    at org.apache.spark.launcher.Main.main(Main.java:74)

Additional information:

(thunder)$ ~ which thunder
/Users/naegelejd/.venvs/thunder/bin/thunder
(thunder)$ ~ echo $SPARK_HOME
/usr/local/spark-1.4.0-bin-hadoop1
(thunder)$ ~ python -V
Python 2.7.10

[naegelejd]

Ah it looks like this is fixed on the master branch, just waiting for an update to PyPI. The current "Quickstart" instructions in README.md are incorrect since the version of thunder on PyPI does not work out-of-the-box with the latest version of Spark.

[freeman-lab]

Thanks for the reminder on this! I just pushed a 0.5.1 maintenance release to PyPi including this fix.