Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
BUILDING Spark requires Scala 2.8. This version has been tested with 2.8.0.final. To build and run Spark, you will need to have Scala's bin in your $PATH, or you will need to set the SCALA_HOME environment variable to point to where you've installed Scala. Scala must be accessible through one of these methods on Mesos slave nodes as well as on the master. To build Spark and the example programs, run make. To run one of the examples, use ./run <class> <params>. For example, ./run SparkLR will run the Logistic Regression example. Each of the example programs prints usage help if no params are given. All of the Spark samples take a <host> parameter that is the Mesos master to connect to. This can be a Mesos URL, or "local" to run locally with one thread, or "local[N]" to run locally with N threads. Tip: If you are building Spark and examples repeatedly, export USE_FSC=1 to have the Makefile use the fsc compiler daemon instead of scalac. CONFIGURATION Spark can be configured through two files: conf/java-opts and conf/spark-env.sh. In java-opts, you can add flags to be passed to the JVM when running Spark. In spark-env.sh, you can set any environment variables you wish to be available when running Spark programs, such as PATH, SCALA_HOME, etc. There are also several Spark-specific variables you can set: - SPARK_CLASSPATH: Extra entries to be added to the classpath, separated by ":". - SPARK_MEM: Memory for Spark to use, in the format used by java's -Xmx option (for example, 200m meams 200 MB, 1g means 1 GB, etc). - SPARK_LIBRARY_PATH: Extra entries to add to java.library.path for locating shared libraries. - SPARK_JAVA_OPTS: Extra options to pass to JVM. Note that spark-env.sh must be a shell script (it must be executable and start with a #! header to specify the shell to use).