Transform should run with spark.kryo.requireRegistration=true #1136

Closed
fnothaft opened this Issue Aug 27, 2016 · 2 comments

Comments

Projects
None yet
2 participants
@fnothaft
Member

fnothaft commented Aug 27, 2016

Ran into this when trying to hunt down a stack overflow during serialization with Kryo's FieldSerializer inside the indel realigner, where all serialization should use custom serializers. Anyways, I'm running into a variety of bugs:

  • scala.reflect.ManifestFactory$ClassTypeManifest.class during MarkDups

@fnothaft fnothaft added the bug label Aug 27, 2016

@fnothaft fnothaft self-assigned this Aug 27, 2016

fnothaft added a commit to fnothaft/adam that referenced this issue Aug 28, 2016

[ADAM-1136] Transform runs successfully with kryo registration required
Resolves #1136. Rewrote IndelRealignmentTarget/ZippedTargetSet serializers to
resolve potential StackOverflowException.
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 29, 2016

Member

What's the best way to test this change?

adam-submit --conf spark.kryo.requireRegistration=true -- transform -realign_indels ...
Member

heuermh commented Aug 29, 2016

What's the best way to test this change?

adam-submit --conf spark.kryo.requireRegistration=true -- transform -realign_indels ...
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Aug 29, 2016

Member

@heuermh that's correct.

Member

fnothaft commented Aug 29, 2016

@heuermh that's correct.

@heuermh heuermh closed this in #1138 Aug 30, 2016

heuermh added a commit to heuermh/adam that referenced this issue Sep 27, 2016

[ADAM-1136] Transform runs successfully with kryo registration required
Resolves #1136. Rewrote IndelRealignmentTarget/ZippedTargetSet serializers to
resolve potential StackOverflowException.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment