[SPARK-6551][PYSPARK] Incorrect aggregate results if seqOp(...) mutates its first argument

[megatron-me-uk]

Implement a test for SPARK-6551.

[srowen]

(Can you add a little more to the title, like what the underlying JIRA is about?)

[megatron-me-uk]

I believe that this will now resolve the issues reported in SPARK-6551 and test for regression.

[JoshRosen]

This comment needs to be updated.

[JoshRosen]

Or removed.

[SparkQA]

Test build #1363 has finished for PR 7965 at commit 140f050.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class BlockMatrix(DistributedMatrix):

[JoshRosen]

/cc @davies, I think you had some comments regarding whether zeroValue could be copied over at #7378?

[davies]

This bug is fixed by #7378 (merged into 1.4 and 1.5) , Could you send a patch against 1.3 branch (remove the unrelated changes)?

[megatron-me-uk]

#7378 has been merged into master and fails half of the tests that I have added in this pull request. The returned result is correct however the zeroValue value supplied is changed. Is it desirable to change the zeroValuethat is supplied to rdd.aggregate?

[davies]

@megatron-me-uk I think it's reasonable to change the zeroValue once you provide a mutable function, for example, the builtin function reduce also works in this way.

If we change to reply on deepcopy, it will not work in some cases (for example, object from C extension).

[megatron-me-uk]

OK good point, I guess I will close this pull request then. I think the best way to resole this jira issue is to backport #7378 or just recommend an upgrade to 1.4.