The most basic of Apache Spark applications to demo spark-submit against a local cluster.
This application is, at its core, a modification of the example that can be found at: http://spark.apache.org/docs/latest/quick-start.html#standalone-applications
I will just be tossing in a couple of small tweaks to make things just a bit more interesting and relevant.
Currently, my blog post for setting up a local cluster lives at: http://spark-fu.blogspot.com/2014/11/running-local-apache-spark-cluster.html
Follow the setup instructions to get your local cluster running (if you don't have a local cluster already and the do the following.
Step 1: Move the file tenzingyatso.txt to a known location on your file system (E.g. /tmp/tenzingyatso.txt)
Step 2: Modify SuperSimple.scala so the path to tenzingyatso.txt is correct for your system.
From: val compassionFile = "/home/bkarels/tenzingyatso.txt"
To: val compassionFile = "/tmp/tenzingyatso.txt"
Step 3: From the root of this project run package from within SBT:
*** Take note of where the application jar is written ***
[info] Packaging /home/bkarels/dev/super-simple-spark-app/target/scala-2.10/super-simple-spark-app_2.10-0.1.jar ...
[info] Done packaging.
Step 4: Since this has been designed to run against a local cluster, navigate to your $SPARK_HOME and use spark-submit to send the application to your cluster:
(example) [bkarels@ahimsa spark_1.1.0]$ ./bin/spark-submit --class com.bradkarels.spark.simple.SuperSimple --master spark://127.0.0.1:7077 /home/bkarels/dev/super-simple-spark-app/target/scala-2.10/super-simple-spark-app_2.10-0.1.jar ... Talks of peace: 3 Speaks of love: 2 [bkarels@ahimsa spark_1.1.0]$
spark-submit has many more options not covered here - this was just the most basic of example to get you up and running using spark-submit to a local Spark cluster. Please vist http://spark.apache.org to learn more.