A example skeleton for an application built on top of KeystoneML
Shell Scala
Switch branches/tags
Nothing to show
Clone or download
etrain Merge pull request #2 from etrain/v0.4fix
Updating example application for v0.4
Latest commit 3d42dd2 Mar 6, 2017
Permalink
Failed to load latest commit information.
bin example skeleton Oct 7, 2015
project example skeleton Oct 7, 2015
sbt example skeleton Oct 7, 2015
src/main Updating example application for v0.4 Mar 3, 2017
.gitignore example skeleton Oct 7, 2015
README.md Create README.md Oct 8, 2015
build.sbt Updating example application for v0.4 Mar 3, 2017

README.md

keystone-example

An example skeleton building applications on top of KeystoneML.

To Run This Example:

Launch a KeystoneML cluster using the provided scripts according to these instructions.

Once the cluster is up, ssh onto the master node and execute these commands:

# Build keystone-example
cd /mnt
git clone https://github.com/amplab/keystone-example.git
cd keystone-example
sbt/sbt assembly
~/spark-ec2/copy-dir target/

# Get the data
wget http://qwone.com/~jason/20Newsgroups/20news-bydate.tar.gz
tar -xvzf 20news-bydate.tar.gz

# Copy to HDFS
/root/ephemeral-hdfs/bin/hadoop fs -copyFromLocal 20news-bydate-train/ /data/
/root/ephemeral-hdfs/bin/hadoop fs -copyFromLocal 20news-bydate-test/ /data/

# Run the pipeline
export SPARK_HOME=/root/spark
export KEYSTONE_HOME=/root/keystone
KEYSTONE_MEM=4g ./bin/run-pipeline.sh \
  pipelines.ExamplePipeline \
  --trainLocation /data/20news-bydate-train \
  --testLocation /data/20news-bydate-test