Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adam-shell does not allow additional jars via Spark jars argument #1349

Closed
heuermh opened this issue Jan 12, 2017 · 6 comments
Closed

adam-shell does not allow additional jars via Spark jars argument #1349

heuermh opened this issue Jan 12, 2017 · 6 comments
Labels
Milestone

Comments

@heuermh
Copy link
Member

@heuermh heuermh commented Jan 12, 2017

If I have this right, adam-shell does not allow additional jars via Spark --jars argument

$ adam-shell --jars my-jar.jar
...
scala> import org.bdgenomics.adam.rdd.ADAMContext
<console>:24: error: object bdgenomics is not a member of package org
       import org.bdgenomics.adam.rdd.ADAMContext
                  ^

adam-shell uses --jars to specify the ADAM cli jar

# submit the job to Spark
"$SPARK_SHELL" \
    --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
    --conf spark.kryo.registrator=org.bdgenomics.adam.serialization.ADAMKryoRegistrator \
    --jars ${ADAM_CLI_JAR} \
    "$@"
@fnothaft
Copy link
Member

@fnothaft fnothaft commented Jan 12, 2017

IIRC, you can only have --jarsonce, and first one wins.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Jan 12, 2017

Right, so how do we fix it? :)

@fnothaft
Copy link
Member

@fnothaft fnothaft commented Jan 12, 2017

Well, I can't promise an answer that you'll like, but I'd say either we mark that as a won't fix, or we parse out a --jars argument in the bin/adam-submit script.

@fnothaft
Copy link
Member

@fnothaft fnothaft commented May 12, 2017

I am labeling this as a won't fix for now. @heuermh please review and reopen if you disagree.

@fnothaft fnothaft closed this May 12, 2017
@heuermh
Copy link
Member Author

@heuermh heuermh commented May 12, 2017

Yes, this is fine to close. I've updated the docs downstream to use

$ spark-shell \
    --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
    --conf spark.kryo.registrator=MyKryoRegistrator \
    --jars my-fat-jar.jar

instead of adam-submit.

@fnothaft
Copy link
Member

@fnothaft fnothaft commented May 12, 2017

Thanks @heuermh!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.