[SPARK-3971] [MLLib] [PySpark] hotfix: Customized pickler should work in cluster mode

[davies]

Customized pickler should be registered before unpickling, but in executor, there is no way to register the picklers before run the tasks.

So, we need to register the picklers in the tasks itself, duplicate the javaToPython() and pythonToJava() in MLlib, call SerDe.initialize() before pickling or unpickling.

[falaki]

Thanks a lot.

[SparkQA]

QA tests have started for PR 2830 at commit 0f02050.

• This patch merges cleanly.

[SparkQA]

QA tests have started for PR 2830 at commit 0f02050.

• This patch merges cleanly.

[SparkQA]

QA tests have started for PR 2830 at commit 6b94e15.

• This patch merges cleanly.

[mengxr]

LGTM. Thanks for fixing it! Waiting for Jenkins.

[SparkQA]

QA tests have finished for PR 2830 at commit 0f02050.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have finished for PR 2830 at commit 0f02050.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21815/
Test PASSed.

[SparkQA]

QA tests have started for PR 2830 at commit 0c85fb9.

• This patch merges cleanly.

[davies]

@mengxr @falaki it had passed all the tests, the last two commits are just refactor, I think it's ready to merge.

[mengxr]

let's wait for the last build to finish.

[SparkQA]

QA tests have finished for PR 2830 at commit 6b94e15.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21816/
Test PASSed.

[SparkQA]

QA tests have finished for PR 2830 at commit 0c85fb9.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21821/
Test FAILed.

[davies]

@mengxr The failed case is not related.

[mengxr]

Yes, that's a known flakey test. I've merged this into master. Thanks!

[mengxr]

test this please

[SparkQA]

QA tests have started for PR 2830 at commit 0c85fb9.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2830 at commit 0c85fb9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21823/
Test PASSed.