ICE 2

Link to source code

Team Members:

Name: Avni Mehta, Class Id: 11

Name: (Trinada) Raji Muppala, Class Id: 14

Objective:

Write a spark program to group the words in a given text file based on their starting letters.

Input:

Output:

Code Screenshot

Code Explaination

1. After importing the libraries, we set the spark configuration and spark context.
2. Data from text file is read and stored into a variable.
3. The following transformations are performed on the data: a. FlatMap - to flatten and reshape the input b. Split(by space) and Distinct - self explainatory c. Map - to create value pair (key - first char of the word , value - word). d. GroupBy - group the key which is first letter and all the words for the letter e. mapValues(.map(._2).mkString(",")) - to remove Compact Buffer from output.
4. Then the transformed data is stored in folder ICE2Output in the format - letter, ()

Contritbution of Team Members:

100% for both since both team members completed the ICE individually.