Skip to content

[SPARK-28958][ML][PYSPARK][WIP] pyspark.ml function parity#25662

Closed
zhengruifeng wants to merge 4 commits intoapache:masterfrom
zhengruifeng:ml_sync
Closed

[SPARK-28958][ML][PYSPARK][WIP] pyspark.ml function parity#25662
zhengruifeng wants to merge 4 commits intoapache:masterfrom
zhengruifeng:ml_sync

Conversation

@zhengruifeng
Copy link
Contributor

What changes were proposed in this pull request?

1, remove setters generated by _shared_params_code_gen.py;
2, add common abstract classes like the side side, such as JavaPredictor/JavaClassificationModel/JavaProbabilisticClassifier;
3, for each alg, add its param trait, such as LinearSVCParams;
4, since sharedParam do not have setters, we need to add them in right places;

Why are the changes needed?

yes, some methods in estimators and models in the py side are changed

How was this patch tested?

I test them in pyspark repl for now

@zhengruifeng
Copy link
Contributor Author

zhengruifeng commented Sep 3, 2019

Since I remove the setters generated by _shared_params_code_gen.py, it seems that all algorithms(classification/regression/clustering/als/features/fpm/etc) needed to be updated in one batch.
This PR is now incomplete, and only modify classification&regression for now, because in this way I need to change too many place in a single PR. I wander if there is a better way to do this.

@zhengruifeng
Copy link
Contributor Author

I just test the modification in pyspark repl, and record the updated places in the attachment in the ticket

@SparkQA
Copy link

SparkQA commented Sep 3, 2019

Test build #110039 has finished for PR 25662 at commit dc1e65e.

  • This patch fails RAT tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zhengruifeng
Copy link
Contributor Author

This PR is too large, I need to divide it into small ones in some way.

@zhengruifeng zhengruifeng deleted the ml_sync branch September 4, 2019 02:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants