This is a simple word count job written in Scala for the Spark spark cluster computing platform, with instructions for running on [Amazon Elastic MapReduce] emr in non-interactive mode. The code is ported directly from Twitter's [
WordCountJob] wordcount for Scalding.
host> git clone https://github.com/snowplow/spark-example-project host> cd spark-example-project host> vagrant up && vagrant ssh guest> cd /vagrant guest> sbt assembly
The 'fat jar' is now available as:
assembly command above runs the test suite - but you can also run this manually with:
$ sbt test <snip> [info] + A WordCount job should [info] + count words correctly [info] Passed: : Total 1, Failed 0, Errors 0, Passed 1, Skipped 0
Running on Amazon EMR
- An AWS CLI profile, e.g. spark
- An Amazon S3 bucket, e.g. spark-example-project-your-name
- A EC2 keypair, e.g. spark-ec2-keypair
- A VPC public subnet, e.g. subnet-3dc2bd2a
Make sure you have assembled the jarfile (see above).
Upload and run
guest> inv upload spark spark-example-project-your-name guest> inv run_emr spark spark-example-project-your-name spark-ec2-keypair subnet-3dc2bd2a
You can now monitor the running EMR jobflow in the AWS Elastic MapReduce UI.
Once the job has completed, you should see a folder structure like this in your output bucket:
results | +- _SUCCESS +- part-00000 +- part-00001 +- part-00002 +- part-...
Download the files and check that one file contains:
while another file contains:
Running on your own Spark cluster
If you have successfully run this on your own Spark cluster, we would welcome a pull-request updating the instructions in this section.
Fork this project and adapt it into your own custom Spark job.
To invoke/schedule your Spark job on EMR, check out:
- [Spark Plug] spark-plug for Scala
- Elasticity elasticity for Ruby
- Boto boto for Python
- Lemur lemur for Clojure
- Change output from tuples to TSV ([#2] issue-2)
Copyright and license
Copyright 2013-2015 Snowplow Analytics Ltd.
Licensed under the [Apache License, Version 2.0] license (the "License"); you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.