No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
data
src/main
.gitignore
newjson.json
pom.xml
readme.txt

readme.txt

Install and fire up the Sandbox using the instructions here: http://maprdocs.mapr.com/home/SandboxHadoop/c_sandbox_overview.html. 

Use an SSH client such as Putty (Windows) or Terminal (Mac) to login. See below for an example:
use userid: user01 and password: mapr.

For VMWare use:  $ ssh user01@ipaddress 

For Virtualbox use:  $ ssh user01@127.0.0.1 -p 2222 

You can build this project with Maven using IDEs like Eclipse or NetBeans, and then copy the JAR file to your MapR Sandbox, or you can install Maven on your sandbox and build from the Linux command line, 
for more information on maven, eclipse or netbeans use google search. 

After building the project on your laptop, you can use scp to copy your JAR file from the project target folder to the MapR Sandbox:

use userid: user01 and password: mapr.
For VMWare use:  $ scp  nameoffile.jar  user01@ipaddress:/user/user01/. 

For Virtualbox use:  $ scp -P 2222 nameoffile.jar  user01@127.0.0.1:/user/user01/.  

Copy the data file from the project data folder to the sandbox using scp to this directory /user/user01/data/ekg.csv on the sandbox:

For Virtualbox use:  $ scp -P 2222 data/ekg.csv  user01@127.0.0.1:/user/user01/data/. 

This example runs on MapR 5.2 with Spark 2.0.1 . If running on the sandbox you need to upgrade to MEP 2.0,  Spark  2.0.1 
http://maprdocs.mapr.com/home/UpgradeGuide/UpgradingEcoPacks.html
http://maprdocs.mapr.com/home/Spark/Spark_IntegrateMapRStreams.html

By default, Spark classpath doesn't contain  spark-streaming-kafka-0-9_2.11.jar and spark-streaming-kafka-producer_2.11-2.0.1-mapr-1611.jar 
which you can get here:  
http://repository.mapr.com/nexus/content/groups/mapr-public/org/apache/spark/spark-streaming-kafka-producer_2.11/2.0.1-mapr-1611/spark-streaming-kafka-producer_2.11-2.0.1-mapr-1611.jar
http://repository.mapr.com/nexus/content/groups/mapr-public/org/apache/spark/spark-streaming-kafka-0-9_2.11/2.0.1-mapr-1611/spark-streaming-kafka-0-9_2.11-2.0.1-mapr-1611.jar
There are 3 ways:
1. Build your application with spark-streaming-kafka-0-9_2.11.jar as dependencies. 
2. Manually download the jars and put to SparkHome/jars folder.
3. Manually download the jars and pass the path to spark submit script with --jars option

Step 0: Create the topics to read from (ekgs) and write to (ekgp)  in MapR streams:

maprcli stream create -path /user/user01/stream -produceperm p -consumeperm p -topicperm p
maprcli stream topic create -path /user/user01/stream -topic ekg -partitions 1
maprcli stream topic create -path /user/user01/stream -topic ekgp -partitions 1

to get info on the ekgs topic :
maprcli stream topic info -path /user/user01/stream -topic ekg

to delete
maprcli stream topic delete -path /user/user01/stream -topic ekgp 
____________________________________________________________________

Step 1:  Run the Spark k-means program which will create and save the machine learning model: 

spark-submit --class com.sparkml.ekg.ClusterEkg --master local[1]  streaming-ekg-1.0.jar 

you can also copy paste  the code into the spark shell 

$spark-shell --master local[2]

There is also a notebook file, in the notebooks directory, which you can import and run in Zeppelin, or view in a Zeppelin viewer
    
 - For Yarn you should change --master parameter to yarn-client - "--master yarn-client"

____________________________________________________________________

After creating and saving the k-means model you can run the Streaming code:

Step 2: Run the MapR Streams Java producer to produce messages, run the Java producer with the topic (ekg) and data file arguments (toReconstruct.tsv.csv):

java -cp streaming-ekg-1.0.jar:`mapr classpath` com.streamskafka.ekg.MsgProducer /user/user01/stream:ekg /user/user01/data/toReconstruct.tsv

To run the MapR Streams Java consumer to see what was published :

java -cp streaming-ekg-1.0.jar:`mapr classpath` com.streamskafka.ekg.MsgConsumer /user/user01/stream:ekg


Step 4:  Run the Spark Consumer Producer with the arguments: path to model, subscribe topic, publish topic :
(in separate consoles if you want to run at the same time)

spark-submit --class com.sparkkafka.ekg.SparkKafkaConsumerProducer --master local[1] \
 streaming-ekg-1.0.jar /user/user01/data/anomaly-detection-master /user/user01/stream:ekg /user/user01/stream:ekgp

____________________________________________________________________



To run the MapR Streams Java consumer  with the topic to read from (ekgp):

java -cp streaming-ekg-1.0.jar:`mapr classpath` com.streamskafka.ekg.MsgConsumer /user/user01/stream:ekgp

_________________________________________________________________________________  



_________________________________________________________________________________

to delete the  topic after using :
maprcli stream topic delete -path /user/user01/stream -topic ekgp