how to config kryo.registrator programmatically #1245

Closed
Fei-Guang opened this Issue Nov 7, 2016 · 1 comment

Comments

Projects
None yet
1 participant
@Fei-Guang

Fei-Guang commented Nov 7, 2016

hello,

it has no problem to run the application by config via command line
SparkSubmit --conf spark.kryo.registrator=org.bdgeno mics.adam.serialization.ADAMKryoRegistrator --conf spark.serializer=org.apache.s park.serializer.KryoSerializer --class org.apache.spark.examples.SparkPi .\adam2 -1.jar

but it report java.io.NotSerializableException: org.bdgenomics.formats.avro.AlignmentRecord
when run
SparkSubmit --class org.apache.spark.examples.SparkPi .\adam2-1.jar

but i have configured it programmatically like the following:

val spark = SparkSession.builder.master("local[*]").appName("Anno BDG") .getOrCreate() //set new runtime options

     spark.conf.set("spark.kryo.registrator", "org.bdgenomics.adam.serialization.ADAMKryoRegistrator")
    spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
@Fei-Guang

This comment has been minimized.

Show comment
Hide comment
@Fei-Guang

Fei-Guang Nov 7, 2016

SparkSession.builder.config("spark.kryo.registrator", "org.bdgenomics.adam.serialization.ADAMKryoRegistrator")

SparkSession.builder.config("spark.kryo.registrator", "org.bdgenomics.adam.serialization.ADAMKryoRegistrator")

@Fei-Guang Fei-Guang closed this Nov 7, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment