2) Spark mapReduce & Actions and Transformations & find top 10 Authors

Spark MapReduce & Top 10 Authors

val top10Authors =
    rdd.map(x => (x.getString("author"), 1)) //use map fn to get author and create tuple(author,no of book=1)
    .reduceByKey(_ + _) //create condensed tuple (author,total no of books) where key is author 
    .sortBy(_._2, ascending = false) //descending sort by number of books
    .take(10) // take top 10 tuples alone 

  top10Authors.foreach(println) //print the top 10 authors

rdd.map(x => (x.getString("author"), 1)) maps each RDD element to tuple of (AuthorName, books=1)

.reduceByKey(_ + _) resolves to

.reduceByKey((x, y) => x + y)

So this is just like a GroupBy operation in SQL. reduce By Key works in following steps. for a given mapped data

('a', 1)
('b', 1)
('a', 1)
('c', 1)

The data is first converted to

('a',[1, 1])
('b', [1])
('c', [1])

and then

('a', 2)
('b', 1)
('c', 1)

so the sortBy(_._2, ascending = false) sorts the elements based on the second element of the tuple in descending order.

Actions and Transformations

take is an action in spark, while map and reduce are transformations. No computation would occur during transformations. Spark just saves the instructions that need to be executed on the RDD data and once an action occurs, it computes all the instructions and returns the data. This is known as lazy evaluation. This is actually scala feature.

So when take is called it computes the instructions on the RDD data and returns only the amount specified by the take argument.