Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate which classes require registration with Kryo #155

Open
tovbinm opened this issue Oct 12, 2018 · 0 comments
Open

Investigate which classes require registration with Kryo #155

tovbinm opened this issue Oct 12, 2018 · 0 comments
Assignees
@marcovivero
Labels
investigation required

Comments

@tovbinm
Copy link
Collaborator

tovbinm commented Oct 12, 2018 •
edited
Loading

Problem
We might have some unregistered classes with Kryo that compromise performance.

Solution
Investigate which classes require registration with Kryo by set spark.kryo.registrationRequired=true in TestSparkContext. Then go over failing tests and see if any classes need to be registered in OpKryoRegistrator.

Additional context
Only do this once we upgrade to Spark 2.3 - #44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@marcovivero marcovivero

Labels
investigation required
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tovbinm @marcovivero