Assignments

A0. MapReduce

MapReduce review ETL

Useful references:

• Configuring Hadoop and running MapReduce applications
• Calculating average ratings and using custom counters
• Sorted ratings with Partitioner, SortComparator and GroupingComparator

A1. Spark RDD API

Spark RDD API

Useful references:

• Introduction to PySpark RDD API
• Spark and Processing Customer Reviews

A2. Spark DataFrame API

Trip Analysis

Useful references:

• Bike Trips and Spark DataFrames
• Spark SQL Functions
• Introduction to Graph Analysis with Spark GraphFrames

A3. Spark Streaming

• Spark Streaming with Twitter API
• Spark Streaming with Telegram API

Useful references:

• Introduction to Spark Streaming
• Spark Streaming and Twitter API
• Introduction to Tweet JSON
• Standard streaming API request parameters

A4. Spark MLlib

Recommendation System

Useful references:

• Rating prediction using Item-Based CF and Matrix Factorization based on ALS
• Regression with Spark MLlib CrossValidator
• MovieLens: dataset overview