In this tutorial you learn how to deploy an Apache Spark streaming application on Cloud Dataproc and process messages from Cloud Pub/Sub in near real-time. The system you build in this scenario generates thousands of random tweets, identifies trending hashtags over a sliding window, saves results in Cloud Datastore, and displays the results on a web page.
Please refer to the related article for all the steps to follow in this tutorial:
https://cloud.google.com/solutions/using-apache-spark-dstreams-with-dataproc-and-pubsub
Contents of this repository:
http_function: Javascript code for the HTTP function deployed on Cloud Functions.spark: Scala code for the Apache Spark streaming application.tweet-generator: Python code for the randomized tweet generator.
To run the tests:
cd spark
mvn test