[SPARK-24947] [Core] aggregateAsync and foldAsync #21971
Conversation
See the description in the [Jira ticket](https://issues.apache.org/jira/browse/SPARK-24947). This contribution is my original work (inspired by similar methods in the Spark code) and I license this work to Spark under the Apache License 2.0.
| * @see [[RDD.aggregate]] which is the synchronous version of this method. | ||
| */ | ||
| def aggregateAsync[U](zeroValue: U)(seqOp: (U, T) => U, combOp: (U, U) => U): FutureAction[U] = | ||
| self.withScope { |
There was a problem hiding this comment.
In the synchronous version of aggregate, the zeroValue is cloned, which requires adding an implicit ClassTag[U] argument. I didn't really understand the motivation for that, so I didn't do it here, but I was hoping that someone who understood the cloning could let me know here.
There was a problem hiding this comment.
IMHO one reason could be the rule of fail fast and early. As the cloning uses serialisation/deserialisation which anyway needed for sending the neutral element to the executors this way serialisation of the neutral element is tested when the operator is specified not during the lazy execution in the middle of a long chain of operations.
| self.withScope { | ||
| val cleanSeqOp = self.context.clean(seqOp) | ||
| val cleanCombOp = self.context.clean(combOp) | ||
| val combBinOp = new BinaryOperator[U] { |
There was a problem hiding this comment.
Is there a cleaner way to integrate with BinaryOperator before Scala 2.12?
|
Can one of the admins verify this patch? |
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
See the description in the Jira ticket.
This contribution is my original work (inspired by similar methods in
the Spark code) and I license this work to Spark under the Apache
License 2.0.
What changes were proposed in this pull request?
Add
aggregateAsyncandfoldAsyncmethods toAsyncRDDActions.How was this patch tested?
Unit tests (included in PR).