add foldLeftByKey to PairRDDFunctions for reduce algorithms that by key ...

[koertkuipers]

...need to process values in a particular order

see:
https://issues.apache.org/jira/browse/SPARK-3655

this is the second of 2 competing pullreqs that try to address this issue. this one does so without making changes to core spark sorting routines. it is based on this suggestion by patrick wendell:

1. Map your RDD[(K, V)] to an RDD[((K, V), null)]
2. Write a custom partitioner that partitions based only on the K component of the key.
3. Call repartitionAndSortWithinPartition with your custom partitioner
4. Map the RDD back into RDD[(K, V)]

the downsides of this approach are that

1. a little more data goes through the shuffle (one extra object per row), i am not sure if this matters at all
2. the sorting by value is not generalized

the upside is that it's a much simpler and more self-contained change than the other pullreq.

[AmplabJenkins]

Can one of the admins verify this patch?

[zsxwing]

Why not an implicit Ordering?

def foldLeftByKey(zeroValue: U, partitioner: Partitioner)(func: (U, V) => U)(implicit vt: ClassTag[U], valueOrdering: Ordering[V]): RDD[(K, U)]

[koertkuipers]

Does this make it harder for the user to provide an ordering other than the natural ordering?

[zsxwing]

I suggest a new name for foldLeftByKey. The semantics of foldLeft means aggregate the values from left to right which does not imply sorting the values.

[koertkuipers]

Hey @zsxwing,
In Scala Seq the order in which the values get processed in foldLeft is well defined.
But can we make any assumptions at all about the ordering of the values if you do not sort them in Spark? And if not, is foldLeft without sorting still useful?

If so, i guess we can make the sorting optional. Or rename this function to make it clear it sorts.

[koertkuipers]

i am going to close this pullreq. i hope to pick up foldLeft later again (together with a proper java version), but for SPARK-3655 the focus for now is on:
#3632