- The goal is to produce a template for a list of 10,000 users which will be used for a mass email campaign.
- Please assume that all of these datasets are large enough that they wouldn't fit into memory on one machine.
- Please use Scala and Spark RDD (not DataFrames or SQL) as we use Spark and Scala in production for our Data Science and Machine Learning applications.
- Although the data is small in this exercise, we want you to assume that data is large and as such you should use a distributed data processing platform (Spark).
- This exercise is designed to test your design and coding abilities for preparing and transforming a large data set.
- Parse the files as necessary
- The top 10,000 customers will be determined by the total amount of money they have spent in the year 2016
- We cannot reach out to a customer via an email address that is blacklisted. However, if a customer has multiple email addresses, and some are on the blacklist while others are not, we want to contact them via all their email addresses that aren't in the blacklist
- Save the campaign in a file with following structure: CustomerId, Customer full name, email list, total transaction amount
- make sure data of customers, transactions, and blacklist located in dirs ./customers, ./transactions, and ./blacklist respectively
- sbt clean; sbt package
- spark-submit --class topcustomers.Driver --master local[4] ./target/scala-2.11/exam_2.11-0.0.1.jar, or spark-submit --class topcustomers.Driver --master spark://localhost:7077 ./target/scala-2.11/exam_2.11-0.0.1.jar
- result is generated at ./output.txt