The most basic of Apache Spark applications to demo spark-submit against a local cluster.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
project
src/main/scala/com/bradkarels/spark/simple
.gitignore
LICENSE.md
README.md
superSimpleSparkApp.sbt
tenzingyatso.txt

README.md

super-simple-spark-app

The most basic of Apache Spark applications to demo spark-submit against a local cluster.

This application is, at its core, a modification of the example that can be found at: http://spark.apache.org/docs/latest/quick-start.html#standalone-applications

I will just be tossing in a couple of small tweaks to make things just a bit more interesting and relevant.

Currently, my blog post for setting up a local cluster lives at: http://spark-fu.blogspot.com/2014/11/running-local-apache-spark-cluster.html

Follow the setup instructions to get your local cluster running (if you don't have a local cluster already and the do the following.

Step 1: Move the file tenzingyatso.txt to a known location on your file system (E.g. /tmp/tenzingyatso.txt)

Step 2: Modify SuperSimple.scala so the path to tenzingyatso.txt is correct for your system.

From: val compassionFile = "/home/bkarels/tenzingyatso.txt"

To: val compassionFile = "/tmp/tenzingyatso.txt"

Step 3: From the root of this project run package from within SBT:

$sbt package

*** Take note of where the application jar is written ***

[info] Packaging /home/bkarels/dev/super-simple-spark-app/target/scala-2.10/super-simple-spark-app_2.10-0.1.jar ...
[info] Done packaging.

> exit

Step 4: Since this has been designed to run against a local cluster, navigate to your $SPARK_HOME and use spark-submit to send the application to your cluster:

(example) [bkarels@ahimsa spark_1.1.0]$ ./bin/spark-submit --class com.bradkarels.spark.simple.SuperSimple --master spark://127.0.0.1:7077 /home/bkarels/dev/super-simple-spark-app/target/scala-2.10/super-simple-spark-app_2.10-0.1.jar ... Talks of peace: 3 Speaks of love: 2 [bkarels@ahimsa spark_1.1.0]$

spark-submit has many more options not covered here - this was just the most basic of example to get you up and running using spark-submit to a local Spark cluster. Please vist http://spark.apache.org to learn more.