Skip to content
Spark app that demonstrates reading and writing data to from MongoDB and BSON files
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src/main/java/com/mongodb/spark/demo updates for mvn artifacts and some logging Sep 18, 2014
.gitignore first pass at Spark Recommender that reads BSON and writes back to Mo… Jun 1, 2014
LICENSE
README.md minor edits Jun 21, 2014
pom.xml updates for mvn artifacts and some logging Sep 18, 2014

README.md

MongoDB-Spark Demo

Prerequisites

To build the MongoDB-Spark demo application, you'll need to have the following:

MongoDB-Hadoop

Note: you must build the MongoDB-Hadoop connector from source for your version of Hadoop and install the core JAR file to your local Maven repo. For example:

$ git clone http://github.com/mongodb/mongo-hadoop.git
$ cd mongo-hadoop
$ ./gradlew jar -Phadoop_version='2.4'
$ mvn install:install-file \
    -Dfile=core/build/lib/mongo-hadoop-core-1.2.1-SNAPSHOT-hadoop_2.4.jar \
    -DgroupId=com.mongodb \
    -DartifactId=hadoop \
    -Dversion=1.2.1-SNAPSHOT \
    -Dpackaging=jar

Spark

Refer to the Spark overview to get started.

Building

To the build the MongoDB-Hadoop demo applications use Maven:

$ mvn package

This will build the demo application and place all of the dependencies in target/lib. If instead you want to build a single jar with all of the dependencies, execute the assembly:single Maven goal:

$ mvn compile assembly:single

Running

$ cd your-spark-directory
$ SPARK_JAR=assembly/target/scala-2.10/spark-assembly-1.0.0-hadoop2.4.0.jar \
  HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop \
  bin/spark-submit --master local --class com.mongodb.spark.demo.Recommender /path/to/demo-1.0-SNAPSHOT.jar \
  --jars /path/to/mongo-java-driver-2.12.2.jar,/path/to/hadoop-1.2.1-SNAPSHOT.jar \
  --executor-memory 4G /movielens/ratings.bson /movielens/users.bson \
  /movielens/movies.bson movielens.predictions

Notes

None at this time.

You can’t perform that action at this time.