Lab 1

Link to source code

Team Members:

Name: Avni Mehta, Class Id: 11

Name: (Trinadha) Raji Muppala, Class Id: 14

Task 1 Objective:

For Movielens dataset, write Spark Transformations and Actions to find the users who have rated more than 25 items.

Input:

The input for this task is movielens data set having 100k records of tab-separated list of user id | item id | rating | timestamp

Movielens dataset

Code

One way:

After importing the libraries, we set the spark configuration and spark context. Data from text file is read and stored into a variable.

An RDD of userids is obtained by mapping the input after splitting by '\t' and getting the first element in the row. The userids RDD is tranformed into a Map of (userids, count of reviews) by countByValue action. The output is filtered where the count > 25 and sorted by asc order.

The output is printed on the console and is also stored into a text file.

Another way:

In below code: Reading file, splitting each line data for ‘\t’ . Applying map transformation to generate RDD with Userid and item details. Applying groupByKey transformation to group data by Userid . Then filtering users where items size is greater than 25. Creating new RDD just user id and number of movies rated

Lastly saving data do text file . Collecting all the data and printing user id , number of movies rated.

Output:

Out is printed on the console as well as stored as a text file

Task 2 Objective:

In GitHub account, create 2 (Source and Documentation) directories and post files in appropriate folders. Include the documentation in the wiki page.

Output:

Task 3 Objective:

In ZenHub, Create a board, 3 milestones, at least 5 issues.

Output:

board:

milestones:

issues:

analytics report:

Contritbution of Team Members:

100% for both since both team members completed the assignment individually.