ch05 - potentially unexpected result from sample code

[dolaameng]

On page 92 in calculating sumSquares, the code

val sumSquares = dataAsArray.fold(
  new Array[Double](numCols)
)(
  (a,b) => a.zip(b).map(t => t._1 + t._2 * t._2)
)

As the RDD.fold requires operator to be communicative, which was violated by asymmetry in the map() function, the result might be different for different number of partitions in RDD.

[srowen]

I think you're right; this should really be aggregate? that is it needs to handle adding together the current sum of squares, and the square of a new value, differently from summing partial sums of squares?

    val sumSquares = dataAsArray.aggregate(
        new Array[Double](numCols)
      )(
        (a, b) => a.zip(b).map(t => t._1 + t._2 * t._2),
        (a, b) => a.zip(b).map(t => t._1 + t._2)
      )

[srowen]

Resolved in source and filed as an erratum.

[dolaameng]

Another way might be just using stats(), even though I think its a little bit slower. The book is a great work. Thanks!

[srowen]

Fair enough, yes; I kind of wanted to show computing these things directly in an FP-oriented way.