Team 3 Capstone: Real-Time Movie Rating Data Pipeline

Introduction

This data pipline will process the Netflix movie real-time rating data, visualize the movie rating trend, and provide a scalable distributed database for real-time query and Machine learning batching processing for movie recommendation.

Team member

Tongzhou Zhou, Bin Shi, Albert, Anthony

Dataset

Netflix Prize data

Architecture

data-ingestion-sinking-layer: Kafka, Kafka Connect
data-transformation-layer: Kafka Streaming
data-storage-layer: Cassandra
data-visualization-layer: NodeJS
data-batch-processing-layer: Spark
cloud-infrastructure: AWS

​ ​ Detailed Schema

Cassandra tables:

CREATE TABLE bittiger.recommendation (
		user_id int PRIMARY KEY,
		recommendations map<text, text>
)

CREATE TABLE bittiger.movielens_movie (
		movie_id int,
		start_time timestamp,
		count int,
		genre text,
		movie_title text,
		rating float,
		users text,
		PRIMARY KEY (movie_id, start_time)
) WITH CLUSTERING ORDER BY (start_time DESC)

Deliverable

Outcome:

	(1)We are going to have a scalable distributed data pipeline for real-time data processing and sinking;
	(2)We are going to provide a scalable distributed database for real-time query like:
		a: Top 10 highest rating movie by a given month
		b: Top 10 rated movie by a given month
		c: Top rated month by a movie
	(3)We are going to build a batch processing layer based on Cassandra and Spark to forecast moive rating.

Weekly milestone:

	Week1: Figure out project architecture, data source, determine the requirements and the functionalities to implement
	Week2: Each team member starts implementing their own module
	Week3: Each team member finishes their own module
	Week4: Testing, report documenting		

Detail Ownerships:

	Tongzhou Zhou: Architecture design, implementation
	Week 1: Set up Kafka Cluster, Cassandra Cluster in AWS (Done)
	Week 2: Starts implement data transformation layer, Set up Kafka Connect to load data from Kafka Cluster to Cassandra Cluster (Done)
	Week 3: Finish implementing data transformation layer: (30% Done)
		(1) Aggregate the data into formats to support the Cassandra data schemas in (2)
		(2) Configure Cassandra data schema to support:
			a: Top 10 highest rating movie by a given month
			b: Top 10 rated movie by a given month
			c: Top rated month by a movie
		Finish S3 set up for batch data sinking.
	Week 4: Starts and finishes unit testing.

	Bin Shi: Data ingestion using kafka; Use machine learning to predict the movie rating
	Week 1: Produce data to kafka [done]
			Send movie title data [done]
			Send rating datea simulating the streaming [done]
	Week 2: Add error/exception handling and more comments in the source code [done]
	Week 3: Study the classification algorithm,and apply to Netflix dataset
	Week 4: Adjust the ML model & Summarize this project

	Albert: Data visualization
	Week 1: Build database to store data.
	Week 2: design analyst algorithm, optimize database if needed.
	Week 3: Collaborate with Anthony to implement the backend for data visualization, and the front end to display data.
	Week 4: Starts and finishes unit testing.

	Anthony: Data visualization
	Week 1: Determine the requirements to implement, get ready to fulfill the Nodejs module features; Data cleaning.(DONE)
	Week 2: Setup backend for data visualization; simple front end to display data.(DONE)
	Week 3: Finish implementing data visualization layer.
	Week 4: Starts and finishes unit testing.

​