how to submit my own driver class via adam-submit? #1289

Closed
Fei-Guang opened this Issue Nov 24, 2016 · 4 comments

Comments

Projects
None yet
2 participants
@Fei-Guang

Fei-Guang commented Nov 24, 2016

i just created a driver as the following:


// scalastyle:off println
package com.ano.adam

import org.apache.spark.sql.SparkSession
import org.bdgenomics.adam.rdd.ADAMContext._


/** Count kmer in the specified format */
object AnnoSp {
  def main(args: Array[String]) {
    val spark = SparkSession.builder
      .config("spark.kryo.registrator", "org.bdgenomics.adam.serialization.ADAMKryoRegistrator")
      .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .config("spark.submit.deployMode", "client")

      //.config("spark.yarn.jars", "")
      .master("local[*]")

and packed it use
jar cvf mydriver1.jar -C classes .\com\ano\adam

how to submit mydriver1.jar to spark via adam-submit?

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Nov 24, 2016

Member

Hello @Fei-Guang, there is a full example in this style at https://github.com/heuermh/adam-examples.

Copy from the build there to configure maven-shade-plugin to create a fat jar and then submit via spark-submit, e.g.

$ spark-submit \
  --master ... \
  --class com.ano.adam.AnnoSp \
  target/annosp.jar
Member

heuermh commented Nov 24, 2016

Hello @Fei-Guang, there is a full example in this style at https://github.com/heuermh/adam-examples.

Copy from the build there to configure maven-shade-plugin to create a fat jar and then submit via spark-submit, e.g.

$ spark-submit \
  --master ... \
  --class com.ano.adam.AnnoSp \
  target/annosp.jar
@Fei-Guang

This comment has been minimized.

Show comment
Hide comment
@Fei-Guang

Fei-Guang Nov 24, 2016

yes, i know how to submit it in a fat jar via spark-submit, but how to do it via adam-submit?

Fei-Guang commented Nov 24, 2016

yes, i know how to submit it in a fat jar via spark-submit, but how to do it via adam-submit?

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Nov 24, 2016

Member

Sorry, I'm not sure exactly what you are saying.

If you want to do it the other way, then you can implement Command, provide your own main that delegates to ADAMMain, and use adam-submit

$ ADAM_MAIN=com.ano.adam.AnnoSpMain \
  adam-submit \
  --jars target/annosp.jar \
  -- \
  anno-sp

Full example in this style at https://github.com/heuermh/adam-commands

Member

heuermh commented Nov 24, 2016

Sorry, I'm not sure exactly what you are saying.

If you want to do it the other way, then you can implement Command, provide your own main that delegates to ADAMMain, and use adam-submit

$ ADAM_MAIN=com.ano.adam.AnnoSpMain \
  adam-submit \
  --jars target/annosp.jar \
  -- \
  anno-sp

Full example in this style at https://github.com/heuermh/adam-commands

@Fei-Guang

This comment has been minimized.

Show comment
Hide comment
@Fei-Guang

Fei-Guang Nov 24, 2016

that's what i want , many thanks

that's what i want , many thanks

@Fei-Guang Fei-Guang closed this Nov 24, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment