[SPARK-13032] [ML] [PySpark] PySpark support model export/import and take LinearRegression as example

[yanboliang]

• Implement MLWriter/MLWritable/MLReader/MLReadable for PySpark.
• Making LinearRegression to support save/load as example. After this merged, the work for other transformers/estimators will be easy, then we can list and distribute the tasks to the community.

cc @mengxr @jkbradley

[SparkQA]

Test build #48300 has finished for PR 10469 at commit 19b07b5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class LinearRegressionModel(JavaModel, MLWritable, TransformerMLReadable):\n * class MLWriter(object):\n * class MLWritable(object):\n * class MLReader(object):\n * class MLReadable(object):\n * java_class = cls._java_loader_class()\n * class TransformerMLReadable(MLReadable):\n * class EstimatorMLReadable(MLReadable):\n

[jkbradley]

@yanboliang I'll take a look at this now. Sorry for the delay!

[jkbradley]

Use directory "/lr_model"?

[jkbradley]

I just added some comments quickly, but let me know if my suggestions are workable. I did not test the suggestions myself.

[SparkQA]

Test build #49987 has finished for PR 10469 at commit 106a2b8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

@yanboliang Thanks for the updates; I'll try to make final comments soon. I left one response in one of the threads above.

[SparkQA]

Test build #50099 has finished for PR 10469 at commit 3a0d6be.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yanboliang]

Unify the construction of Model and Estimator. Model can be instantiated without argument which is used by load.

[jkbradley]

Can you add a note in the doc to explain this?

[jkbradley]

Thanks for the updates! Done with a pass. They are mostly minor comments, except for making MLReadable, MLWritable more general and not specific to Java wrappers.

[yanboliang]

@jkbradley Thanks for your comments! I have made MLReadable and MLWritable more general and not specific to Java wrappers, addressed all comments except for setting the Param.parent. To that issue, I left the inline comment in the threads above.

[SparkQA]

Test build #50182 has finished for PR 10469 at commit 7329124.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class JavaMLWriter(object):
  • class JavaMLReader(object):

[Wenpei]

HI
I raise a common issues here when I start look pyspark. I found there is only one test.py to test basic RDD and spark submit related api. There is no ml & mllib related unit test currently, I knew we just implement wrapper here, but some place may need ut, for example, parameter process.
Just raise what I thought.

Regards.
Wenpei

[jkbradley]

@Wenpei This isn't the right forum for posting comments like that (since hardly anyone will see your comment). I'd recommend identifying missing unit tests and making JIRAs for them. We do have tests.py files under pyspark/ml and pyspark/mllib, so please do check those first before making JIRAs.

[jkbradley]

@yanboliang I hope you don't mind, but I took the liberty of experimenting a bit myself and sending this PR: [https://github.com/yanboliang/pull/4] Please let me know what you think!

Btw, thanks for the generalization-related updates. I guess we'd have to go further (providing MLWriter, MLReader abstract classes) if we wanted to allow Python developers to implement persistence from Python, but we can address that in the future (if anyone requests it). Your changes should help us work towards that though.

[SparkQA]

Test build #50264 has finished for PR 10469 at commit f07ffcb.

• This patch fails Python style tests.
• This patch does not merge cleanly.
• This patch adds the following public classes (experimental):\n * class LinearRegressionModel(JavaModel, MLWritable, MLReadable):\n * class JavaMLWriter(object):\n * class MLWritable(object):\n * class JavaMLReader(object):\n * java_class = cls._java_loader_class(clazz)\n * class MLReadable(object):\n

[SparkQA]

Test build #50267 has finished for PR 10469 at commit 7334be9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yanboliang]

@jkbradley You PR looks good and get merged, thanks!

[jkbradley]

LGTM
Thanks for this PR! This makes it really simple to add persistence.
I'll merge it with master.