FastaConverter.FastaDescriptionLine not kryo-registered #893

Closed
ryan-williams opened this Issue Dec 2, 2015 · 5 comments

Comments

Projects
None yet
3 participants
@ryan-williams
Member

ryan-williams commented Dec 2, 2015

Running some apps with spark.kryo.registrationRequired=true and getting failures due to this.

Pretty sure the fix is just:

kryo.register(classOf[FastaConverter.FastaDescriptionLine])

in ADAMKryoRegistrator

@ryan-williams ryan-williams added the bug label Dec 2, 2015

@ryan-williams

This comment has been minimized.

Show comment
Hide comment
@ryan-williams

ryan-williams Dec 2, 2015

Member

Also ReferenceRegion, which @ipurusho is hitting.

Member

ryan-williams commented Dec 2, 2015

Also ReferenceRegion, which @ipurusho is hitting.

@fnothaft fnothaft added this to the 0.19.0 milestone Jan 7, 2016

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 7, 2016

Member

@ryan-williams did you have a fix for this/code that reproduces the issue? I'm a bit surprised about any issues with ReferenceRegion, can you elaborate?

Member

fnothaft commented Jan 7, 2016

@ryan-williams did you have a fix for this/code that reproduces the issue? I'm a bit surprised about any issues with ReferenceRegion, can you elaborate?

@ryan-williams

This comment has been minimized.

Show comment
Hide comment
@ryan-williams

ryan-williams Jan 7, 2016

Member

I was aggressively auditing my serialization story and running with spark.kryo.registrationRequired=true, and it was pretty trivial to uncover these. I'll think about exactly what would do it, but any join on a ReferenceRegion should do it, I'd think.. anything with a shuffle stage involving records of the types in question that get ser'd...

Member

ryan-williams commented Jan 7, 2016

I was aggressively auditing my serialization story and running with spark.kryo.registrationRequired=true, and it was pretty trivial to uncover these. I'll think about exactly what would do it, but any join on a ReferenceRegion should do it, I'd think.. anything with a shuffle stage involving records of the types in question that get ser'd...

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 7, 2016

Member

Ah, great! I missed that you were running with spark.kryo.registrationRequired=true. This makes perfect sense now; sorry for the confusion.

Member

fnothaft commented Jan 7, 2016

Ah, great! I missed that you were running with spark.kryo.registrationRequired=true. This makes perfect sense now; sorry for the confusion.

fnothaft added a commit to fnothaft/adam that referenced this issue Feb 8, 2016

[ADAM-893] Register missing serializers.
Resolves #893. Registers serializers with Kryo for the ReferenceRegion model.
Additionally, registers the default Java/Scala case class serializer with Kryo
for many missing model classes, and alphabetizes the ADAMKryoRegistration class
registrations for legibility.

@heuermh heuermh modified the milestones: 0.20.0, 0.19.0 Feb 24, 2016

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Apr 28, 2016

Member

Resolved by #1020.

Member

fnothaft commented Apr 28, 2016

Resolved by #1020.

@fnothaft fnothaft closed this Apr 28, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment