Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to submit my own driver class via adam-submit? #1289

Closed
Fei-Guang opened this issue Nov 24, 2016 · 4 comments
Closed

how to submit my own driver class via adam-submit? #1289

Fei-Guang opened this issue Nov 24, 2016 · 4 comments

Comments

@Fei-Guang
Copy link

@Fei-Guang Fei-Guang commented Nov 24, 2016

i just created a driver as the following:


// scalastyle:off println
package com.ano.adam

import org.apache.spark.sql.SparkSession
import org.bdgenomics.adam.rdd.ADAMContext._


/** Count kmer in the specified format */
object AnnoSp {
  def main(args: Array[String]) {
    val spark = SparkSession.builder
      .config("spark.kryo.registrator", "org.bdgenomics.adam.serialization.ADAMKryoRegistrator")
      .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .config("spark.submit.deployMode", "client")

      //.config("spark.yarn.jars", "")
      .master("local[*]")

and packed it use
jar cvf mydriver1.jar -C classes .\com\ano\adam

how to submit mydriver1.jar to spark via adam-submit?

@heuermh
Copy link
Member

@heuermh heuermh commented Nov 24, 2016

Hello @Fei-Guang, there is a full example in this style at https://github.com/heuermh/adam-examples.

Copy from the build there to configure maven-shade-plugin to create a fat jar and then submit via spark-submit, e.g.

$ spark-submit \
  --master ... \
  --class com.ano.adam.AnnoSp \
  target/annosp.jar
@Fei-Guang
Copy link
Author

@Fei-Guang Fei-Guang commented Nov 24, 2016

yes, i know how to submit it in a fat jar via spark-submit, but how to do it via adam-submit?

@heuermh
Copy link
Member

@heuermh heuermh commented Nov 24, 2016

Sorry, I'm not sure exactly what you are saying.

If you want to do it the other way, then you can implement Command, provide your own main that delegates to ADAMMain, and use adam-submit

$ ADAM_MAIN=com.ano.adam.AnnoSpMain \
  adam-submit \
  --jars target/annosp.jar \
  -- \
  anno-sp

Full example in this style at https://github.com/heuermh/adam-commands

@Fei-Guang
Copy link
Author

@Fei-Guang Fei-Guang commented Nov 24, 2016

that's what i want , many thanks

@Fei-Guang Fei-Guang closed this Nov 24, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.