Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PySpark] add picklable SparseMatrix in pyspark.ml.common #14265

Closed
wants to merge 1 commit into from

Conversation

WeichenXu123
Copy link
Contributor

What changes were proposed in this pull request?

add SparseMatrix class whick support pickler.

How was this patch tested?

Existing test.

@WeichenXu123 WeichenXu123 changed the title [PySpark] add picklable SparseMatrix [PySpark] add picklable SparseMatrix in pyspark.ml.common Jul 19, 2016
@SparkQA
Copy link

SparkQA commented Jul 19, 2016

Test build #62536 has finished for PR 14265 at commit 482d7ff.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@WeichenXu123
Copy link
Contributor Author

@srowen
I check the ml.python.MLSerde and it support SparseMatrix pickler and at python side the SparseMatrix constructor also match the pickler. So I think the _picklable_classes forgot to add SparkMatrix, or there is some other reason not to add SparseMatrix to the class list ?

@WeichenXu123
Copy link
Contributor Author

@srowen I guess the _picklable_classes list in ml.linalg.common is copied from mllib.linalg.common so it forgot to add the SparseMatrix which is added later.

@srowen
Copy link
Member

srowen commented Jul 21, 2016

This isn't really my area, but I that sounds plausible

@WeichenXu123
Copy link
Contributor Author

cc @jkbradley Thanks!

@srowen
Copy link
Member

srowen commented Jul 24, 2016

@holdenk or @MLnick do you happen to know if this sounds right?

@yanboliang
Copy link
Contributor

LGTM, merged into master. Thanks.

@asfgit asfgit closed this in 37bed97 Jul 24, 2016
@WeichenXu123 WeichenXu123 deleted the picklable_py branch July 24, 2016 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants