newsStreaming

This package is part of my Insight Project nexTop.

The library runs a Spark Streaming consumer that receives messages posted by the Kafka S3 Producer and sends them into Elasticsearch using the native client library elasticsearch-hadoop.

Build and run instructions

To build the jar:

sbt assembly

Example usage in standalone:

$SPARK_HOME/bin/spark-submit --class insightproject.spark.newsstreaming.NewsStreaming --master \
spark://`hostname`:7077 --jars newsStreaming-assembly-1.0.jar newsStreaming-assembly-1.0.jar \
my_topic my_group_id elastic_destination_index/elastic_doc_type