New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adam-shell does not allow additional jars via Spark jars argument #1349

Closed
heuermh opened this Issue Jan 12, 2017 · 6 comments

Comments

Projects
2 participants
@heuermh
Member

heuermh commented Jan 12, 2017

If I have this right, adam-shell does not allow additional jars via Spark --jars argument

$ adam-shell --jars my-jar.jar
...
scala> import org.bdgenomics.adam.rdd.ADAMContext
<console>:24: error: object bdgenomics is not a member of package org
       import org.bdgenomics.adam.rdd.ADAMContext
                  ^

adam-shell uses --jars to specify the ADAM cli jar

# submit the job to Spark
"$SPARK_SHELL" \
    --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
    --conf spark.kryo.registrator=org.bdgenomics.adam.serialization.ADAMKryoRegistrator \
    --jars ${ADAM_CLI_JAR} \
    "$@"
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 12, 2017

Member

IIRC, you can only have --jarsonce, and first one wins.

Member

fnothaft commented Jan 12, 2017

IIRC, you can only have --jarsonce, and first one wins.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jan 12, 2017

Member

Right, so how do we fix it? :)

Member

heuermh commented Jan 12, 2017

Right, so how do we fix it? :)

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 12, 2017

Member

Well, I can't promise an answer that you'll like, but I'd say either we mark that as a won't fix, or we parse out a --jars argument in the bin/adam-submit script.

Member

fnothaft commented Jan 12, 2017

Well, I can't promise an answer that you'll like, but I'd say either we mark that as a won't fix, or we parse out a --jars argument in the bin/adam-submit script.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft May 12, 2017

Member

I am labeling this as a won't fix for now. @heuermh please review and reopen if you disagree.

Member

fnothaft commented May 12, 2017

I am labeling this as a won't fix for now. @heuermh please review and reopen if you disagree.

@fnothaft fnothaft closed this May 12, 2017

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh May 12, 2017

Member

Yes, this is fine to close. I've updated the docs downstream to use

$ spark-shell \
    --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
    --conf spark.kryo.registrator=MyKryoRegistrator \
    --jars my-fat-jar.jar

instead of adam-submit.

Member

heuermh commented May 12, 2017

Yes, this is fine to close. I've updated the docs downstream to use

$ spark-shell \
    --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
    --conf spark.kryo.registrator=MyKryoRegistrator \
    --jars my-fat-jar.jar

instead of adam-submit.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft May 12, 2017

Member

Thanks @heuermh!

Member

fnothaft commented May 12, 2017

Thanks @heuermh!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment