New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump ADAM to Spark 1.4 #754

Merged
merged 9 commits into from Aug 6, 2015

Conversation

Projects
None yet
5 participants
@laserson
Contributor

laserson commented Aug 5, 2015

Haven't tested this yet. Inserted some inline comments.

@@ -23,10 +23,6 @@ ADAM_REPO="$(cd `dirname $0`/..; pwd)"
CLASSPATH=$("$ADAM_REPO"/bin/compute-adam-classpath.sh)
ADAM_JARS=$("$ADAM_REPO"/bin/compute-adam-jars.sh)
if [ ! -z "$ADDL_JARS" ]; then
ADAM_JARS="$ADAM_JARS","$ADDL_JARS"
fi

This comment has been minimized.

@laserson

laserson Aug 5, 2015

Contributor

Spark has deprecated setting options through env vars. I propose we follow suit.

@laserson

laserson Aug 5, 2015

Contributor

Spark has deprecated setting options through env vars. I propose we follow suit.

This comment has been minimized.

@fnothaft

fnothaft Aug 5, 2015

Member

I am OK with removing ADDL_JARS.

As an aside, how will options be set? Do they need to preface the arguments they are passing to ADAM on the command line with any additional --conf flags?

E.g., does:

cd $ADAM_HOME
export ADAM_OPTS="--conf spark.halt-and-catch-fire=false"
./bin/adam-shell transform myBamFile.bam myAdamFile.adam

become:

cd $ADAM_HOME
./bin/adam-shell --conf spark.halt-and-catch-fire=false transform myBamFile.bam myAdamFile.adam
@fnothaft

fnothaft Aug 5, 2015

Member

I am OK with removing ADDL_JARS.

As an aside, how will options be set? Do they need to preface the arguments they are passing to ADAM on the command line with any additional --conf flags?

E.g., does:

cd $ADAM_HOME
export ADAM_OPTS="--conf spark.halt-and-catch-fire=false"
./bin/adam-shell transform myBamFile.bam myAdamFile.adam

become:

cd $ADAM_HOME
./bin/adam-shell --conf spark.halt-and-catch-fire=false transform myBamFile.bam myAdamFile.adam

This comment has been minimized.

@laserson

laserson Aug 5, 2015

Contributor

I think it would actually become

./bin/adam-shell --conf spark.halt-and-catch-fire=false -- transform myBamFile.bam myAdamFile.adam

Note the --.

@laserson

laserson Aug 5, 2015

Contributor

I think it would actually become

./bin/adam-shell --conf spark.halt-and-catch-fire=false -- transform myBamFile.bam myAdamFile.adam

Note the --.

This comment has been minimized.

@fnothaft

fnothaft Aug 5, 2015

Member

That is OK by me, but it is kind of nice to set options via envars (e.g., if you're submitting jobs via YARN and putting all your config on the command line, your command line can get quite long quite quick). What is the impetus for moving away from envars?

@fnothaft

fnothaft Aug 5, 2015

Member

That is OK by me, but it is kind of nice to set options via envars (e.g., if you're submitting jobs via YARN and putting all your config on the command line, your command line can get quite long quite quick). What is the impetus for moving away from envars?

This comment has been minimized.

@laserson

laserson Aug 5, 2015

Contributor

Not certain. Perhaps it was becoming unwieldy to keep track of where configs were coming from?

Eitherway, perhaps you could run your command like so:

export MY_SPARK_OPTS=...
adam-submit $MY_SPARK_OPTS -- $MY_ADAM_OPTS

Note that this only really affects adam-submit, as the interactive shells don't take any ADAM-specific options, AFAIK.

@laserson

laserson Aug 5, 2015

Contributor

Not certain. Perhaps it was becoming unwieldy to keep track of where configs were coming from?

Eitherway, perhaps you could run your command like so:

export MY_SPARK_OPTS=...
adam-submit $MY_SPARK_OPTS -- $MY_ADAM_OPTS

Note that this only really affects adam-submit, as the interactive shells don't take any ADAM-specific options, AFAIK.

Show outdated Hide outdated bin/adam-pyspark
--conf spark.kryo.referenceTracking=true \
${ADAM_OPTS:- } \
--conf spark.executor.memory=${ADAM_EXECUTOR_MEMORY:-"4g"} \
--driver-memory ${ADAM_DRIVER_MEMORY:-"4g"} \

This comment has been minimized.

@laserson

laserson Aug 5, 2015

Contributor

Just set these options, or pass a conf file.

@laserson

laserson Aug 5, 2015

Contributor

Just set these options, or pass a conf file.

Show outdated Hide outdated bin/adam-submit
# list of jars to ship with spark; trim off the first and last from the CLASSPATH
# TODO: brittle? assumes appassembler always puts the $BASE/etc first and the CLI jar last
# list of jars to ship with spark; trim off the first from the CLASSPATH
# TODO: brittle? assumes appassembler always puts the $BASE/etc first

This comment has been minimized.

@laserson

laserson Aug 5, 2015

Contributor

This comment was never updated along with the relevant change. Speaking of which, will distributing the cli jar with the --jars option stay kosher?

@laserson

laserson Aug 5, 2015

Contributor

This comment was never updated along with the relevant change. Speaking of which, will distributing the cli jar with the --jars option stay kosher?

@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 5, 2015

Contributor

This fixes #659 .

Contributor

laserson commented Aug 5, 2015

This fixes #659 .

@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 5, 2015

Contributor

Also need to update the docs and jenkins script

Contributor

laserson commented Aug 5, 2015

Also need to update the docs and jenkins script

Uri Laserson added some commits Aug 5, 2015

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 5, 2015

Member

As far as I know ADDL_JARS is the only way to make plugins work, e.g. https://github.com/heuermh/adam-plugins

Member

heuermh commented Aug 5, 2015

As far as I know ADDL_JARS is the only way to make plugins work, e.g. https://github.com/heuermh/adam-plugins

@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 5, 2015

Contributor

What about adding the jar to the --jars option?

bin/adam-submit --jars adam-plugins.jar -- plugin com.github.heuermh.adam.plugins.CountAlignments ...
Contributor

laserson commented Aug 5, 2015

What about adding the jar to the --jars option?

bin/adam-submit --jars adam-plugins.jar -- plugin com.github.heuermh.adam.plugins.CountAlignments ...
@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 5, 2015

Contributor

Ok, when I tried vcf2adam on our yarn-managed cluster with Spark 1.4.1 this worked. Ready for merge from my perspective, assuming all the tests pass.

Contributor

laserson commented Aug 5, 2015

Ok, when I tried vcf2adam on our yarn-managed cluster with Spark 1.4.1 this worked. Ready for merge from my perspective, assuming all the tests pass.

@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 5, 2015

Contributor

@heuermh this mitigates the short term need to deal with the spark submit scripts, but I think it's still work investigating using the SparkLauncher library if people think that it's worth it.

Contributor

laserson commented Aug 5, 2015

@heuermh this mitigates the short term need to deal with the spark submit scripts, but I think it's still work investigating using the SparkLauncher library if people think that it's worth it.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Aug 5, 2015

Member

SGTM; I will test this on our internal cluster as well tomorrow.

Member

fnothaft commented Aug 5, 2015

SGTM; I will test this on our internal cluster as well tomorrow.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 5, 2015

Member

This works for me but I have a couple concerns about the user experience:

  • "script-compability" (for lack of a better term) has been broken; all existing ADAM workflows will fail with
$ adam-submit transform -force_load_bam foo.bam foo.adam
Failed to find '--' in argument list.
Usage: adam-submit <spark-args> -- <adam-args>
  • the usage/help screen does not list any <spark-args> or <adam-args>, nor does it show where <command> or <command args> should go
$ ./bin/adam-submit 
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/08/05 10:35:58 INFO ADAMMain: ADAM invoked with args:

     e            888~-_              e                 e    e
    d8b           888   \            d8b               d8b  d8b
   /Y88b          888    |          /Y88b             d888bdY88b
  /  Y88b         888    |         /  Y88b           / Y88Y Y888b
 /____Y88b        888   /         /____Y88b         /   YY   Y888b
/      Y88b       888_-~         /      Y88b       /          Y888b

Usage: adam-submit <spark-args> -- <adam-args>

Choose one of the following commands:

ADAM ACTIONS
               depth : Calculate the depth from a given ADAM file, at each variant in a VCF
...

Perhaps Usage: adam-submit <spark-args> -- <adam-args> <command> <command-args> and an example would help?

Then I don't believe --jars works for the plugin use case, although I might not be using it correctly

$ ./bin/adam-submit --jars adam-plugins_2.10-0.17.1-SNAPSHOT.jar \
  -- plugin com.github.heuermh.adam.plugins.CountAlignments \
  adam-core/src/test/resources/small.sam

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/08/05 10:57:45 INFO ADAMMain: ADAM invoked with args: "plugin" "com.github.heuermh.adam.plugins.CountAlignments" "adam-core/src/test/resources/small.sam"
15/08/05 10:57:45 INFO SparkContext: Running Spark version 1.4.1
...
15/08/05 10:57:46 INFO SparkContext: Added JAR file:.../adam-cli/target/appassembler/repo/commons-cli/commons-cli/1.2/commons-cli-1.2.jar at http://192.168.1.6:49540/jars/commons-cli-1.2.jar with timestamp 1438790266939
...
  no mention of adam-plugins_2.10-0.17.1-SNAPSHOT.jar
...
15/08/05 10:57:47 INFO SparkContext: Added JAR file:.../adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-cli_2.10/0.17.1-SNAPSHOT/adam-cli_2.10-0.17.1-SNAPSHOT.jar at http://192.168.1.6:49540/jars/adam-cli_2.10-0.17.1-SNAPSHOT.jar with timestamp 1438790267091
...
15/08/05 10:57:47 INFO BlockManagerMaster: Trying to register BlockManager
15/08/05 10:57:47 INFO BlockManagerMasterEndpoint: Registering block manager localhost:49541 with 265.1 MB RAM, BlockManagerId(driver, localhost, 49541)
15/08/05 10:57:47 INFO BlockManagerMaster: Registered BlockManager
Exception in thread "main" java.lang.ClassNotFoundException: com.github.heuermh.adam.plugins.CountAlignments
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.bdgenomics.adam.cli.PluginExecutor.loadPlugin(PluginExecutor.scala:72)
    at org.bdgenomics.adam.cli.PluginExecutor.run(PluginExecutor.scala:90)
    at org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:53)
    at org.bdgenomics.adam.cli.PluginExecutor.run(PluginExecutor.scala:66)
    at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:106)
    at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/08/05 10:57:47 INFO SparkContext: Invoking stop() from shutdown hook
...
15/08/05 10:57:47 INFO SparkContext: Successfully stopped SparkContext
15/08/05 10:57:47 INFO Utils: Shutdown hook called
15/08/05 10:57:47 INFO Utils: Deleting directory /private/var/folders/8f/5r73k3zs4dq2j968n458d7j80000gn/T/spark-737b978f-267a-4b74-8951-19a88ef1f7bc
Member

heuermh commented Aug 5, 2015

This works for me but I have a couple concerns about the user experience:

  • "script-compability" (for lack of a better term) has been broken; all existing ADAM workflows will fail with
$ adam-submit transform -force_load_bam foo.bam foo.adam
Failed to find '--' in argument list.
Usage: adam-submit <spark-args> -- <adam-args>
  • the usage/help screen does not list any <spark-args> or <adam-args>, nor does it show where <command> or <command args> should go
$ ./bin/adam-submit 
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/08/05 10:35:58 INFO ADAMMain: ADAM invoked with args:

     e            888~-_              e                 e    e
    d8b           888   \            d8b               d8b  d8b
   /Y88b          888    |          /Y88b             d888bdY88b
  /  Y88b         888    |         /  Y88b           / Y88Y Y888b
 /____Y88b        888   /         /____Y88b         /   YY   Y888b
/      Y88b       888_-~         /      Y88b       /          Y888b

Usage: adam-submit <spark-args> -- <adam-args>

Choose one of the following commands:

ADAM ACTIONS
               depth : Calculate the depth from a given ADAM file, at each variant in a VCF
...

Perhaps Usage: adam-submit <spark-args> -- <adam-args> <command> <command-args> and an example would help?

Then I don't believe --jars works for the plugin use case, although I might not be using it correctly

$ ./bin/adam-submit --jars adam-plugins_2.10-0.17.1-SNAPSHOT.jar \
  -- plugin com.github.heuermh.adam.plugins.CountAlignments \
  adam-core/src/test/resources/small.sam

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/08/05 10:57:45 INFO ADAMMain: ADAM invoked with args: "plugin" "com.github.heuermh.adam.plugins.CountAlignments" "adam-core/src/test/resources/small.sam"
15/08/05 10:57:45 INFO SparkContext: Running Spark version 1.4.1
...
15/08/05 10:57:46 INFO SparkContext: Added JAR file:.../adam-cli/target/appassembler/repo/commons-cli/commons-cli/1.2/commons-cli-1.2.jar at http://192.168.1.6:49540/jars/commons-cli-1.2.jar with timestamp 1438790266939
...
  no mention of adam-plugins_2.10-0.17.1-SNAPSHOT.jar
...
15/08/05 10:57:47 INFO SparkContext: Added JAR file:.../adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-cli_2.10/0.17.1-SNAPSHOT/adam-cli_2.10-0.17.1-SNAPSHOT.jar at http://192.168.1.6:49540/jars/adam-cli_2.10-0.17.1-SNAPSHOT.jar with timestamp 1438790267091
...
15/08/05 10:57:47 INFO BlockManagerMaster: Trying to register BlockManager
15/08/05 10:57:47 INFO BlockManagerMasterEndpoint: Registering block manager localhost:49541 with 265.1 MB RAM, BlockManagerId(driver, localhost, 49541)
15/08/05 10:57:47 INFO BlockManagerMaster: Registered BlockManager
Exception in thread "main" java.lang.ClassNotFoundException: com.github.heuermh.adam.plugins.CountAlignments
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.bdgenomics.adam.cli.PluginExecutor.loadPlugin(PluginExecutor.scala:72)
    at org.bdgenomics.adam.cli.PluginExecutor.run(PluginExecutor.scala:90)
    at org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:53)
    at org.bdgenomics.adam.cli.PluginExecutor.run(PluginExecutor.scala:66)
    at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:106)
    at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/08/05 10:57:47 INFO SparkContext: Invoking stop() from shutdown hook
...
15/08/05 10:57:47 INFO SparkContext: Successfully stopped SparkContext
15/08/05 10:57:47 INFO Utils: Shutdown hook called
15/08/05 10:57:47 INFO Utils: Deleting directory /private/var/folders/8f/5r73k3zs4dq2j968n458d7j80000gn/T/spark-737b978f-267a-4b74-8951-19a88ef1f7bc
@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 5, 2015

Contributor

"script-compability" (for lack of a better term) has been broken; all existing ADAM workflows will fail with

Personally I have no problem with this. We're still in 0.x for exactly this reason, and this seems like a reasonable change to me.

That said, I agree that it looks kinda ugly that we require -- even if not specifying any Spark options. Let me see about changing that.

the usage/help screen does not list any or , nor does it show where or should go

Actually, it's the first thing under the ascii art. But I agree that it's subtle. And anyway, our examples in the README etc should include instances where we also pass Spark options so people can see what it looks like.

Then I don't believe --jars works for the plugin use case, although I might not be using it correctly

Looks right to me. I'm surprised that it doesn't work, as ADDL_JARS just added that list of jars to the --jars arg anyway. Could you rerun with bash -x and post the results (in a gist if it's too large).

Do I need to actually set the CLASSPATH variable in the shell that runs adam-submit? We never did before...

Contributor

laserson commented Aug 5, 2015

"script-compability" (for lack of a better term) has been broken; all existing ADAM workflows will fail with

Personally I have no problem with this. We're still in 0.x for exactly this reason, and this seems like a reasonable change to me.

That said, I agree that it looks kinda ugly that we require -- even if not specifying any Spark options. Let me see about changing that.

the usage/help screen does not list any or , nor does it show where or should go

Actually, it's the first thing under the ascii art. But I agree that it's subtle. And anyway, our examples in the README etc should include instances where we also pass Spark options so people can see what it looks like.

Then I don't believe --jars works for the plugin use case, although I might not be using it correctly

Looks right to me. I'm surprised that it doesn't work, as ADDL_JARS just added that list of jars to the --jars arg anyway. Could you rerun with bash -x and post the results (in a gist if it's too large).

Do I need to actually set the CLASSPATH variable in the shell that runs adam-submit? We never did before...

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 5, 2015

Member

See
https://gist.github.com/heuermh/4105bd5b4ca111979652

Looks like append_to_option.py didn't do the right thing

+ SPARK_ARGS='--jars adam-plugins_2.10-0.17.1-SNAPSHOT.jar --jars /Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-cli/commons-cli/1.2/commons-cli-1.2.jar,/Users
Member

heuermh commented Aug 5, 2015

See
https://gist.github.com/heuermh/4105bd5b4ca111979652

Looks like append_to_option.py didn't do the right thing

+ SPARK_ARGS='--jars adam-plugins_2.10-0.17.1-SNAPSHOT.jar --jars /Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-cli/commons-cli/1.2/commons-cli-1.2.jar,/Users
@laserson

This comment has been minimized.

Show comment
Hide comment
Contributor

laserson commented Aug 5, 2015

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 5, 2015

Member

I think the single quotes throws it off, this works

$ append_to_option.py , --jars /Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-cli/commons-cli/1.2/commons-cli-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-codec/commons-codec/1.4/commons-codec-1.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/slf4j/slf4j-api/1.7.10/slf4j-api-1.7.10.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/log4j/log4j/1.2.17/log4j-1.2.17.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/xerial/snappy/snappy-java/1.1.1.7/snappy-java-1.1.1.7.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/thoughtworks/paranamer/paranamer/2.6/paranamer-2.6.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-io_2.10/0.2.2/utils-io_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-misc_2.10/0.2.2/utils-misc_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/httpcomponents/httpclient/4.3.2/httpclient-4.3.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/httpcomponents/httpcore/4.3.1/httpcore-4.3.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-cli_2.10/0.2.2/utils-cli_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-avro/1.8.1/parquet-avro-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-column/1.8.1/parquet-column-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-common/1.8.1/parquet-common-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-encoding/1.8.1/parquet-encoding-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-hadoop/1.8.1/parquet-hadoop-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-jackson/1.8.1/parquet-jackson-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-format/2.3.0-incubating/parquet-format-2.3.0-incubating.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-metrics_2.10/0.2.2/utils-metrics_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/netflix/servo/servo-core/0.5.5/servo-core-0.5.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/google/code/findbugs/annotations/2.0.0/annotations-2.0.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/scoverage/scalac-scoverage-plugin_2.10/0.99.2/scalac-scoverage-plugin_2.10-0.99.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-io/commons-io/1.3.2/commons-io-1.3.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/bdg-formats/bdg-formats/0.4.0/bdg-formats-0.4.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/avro/avro/1.7.6/avro-1.7.6.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-core_2.10/0.17.1-SNAPSHOT/adam-core_2.10-0.17.1-SNAPSHOT.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/ow2/asm/asm/4.0/asm-4.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/objenesis/objenesis/1.2/objenesis-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/it/unimi/dsi/fastutil/6.4.4/fastutil-6.4.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-scala_2.10/1.8.1/parquet-scala_2.10-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/seqdoop/hadoop-bam/7.0.0/hadoop-bam-7.0.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/github/samtools/htsjdk/1.133/htsjdk-1.133.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/commons/commons-jexl/2.1.1/commons-jexl-2.1.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/tukaani/xz/1.5/xz-1.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/ant/ant/1.8.2/ant-1.8.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/testng/testng/6.8.8/testng-6.8.8.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/beanshell/bsh/2.0b4/bsh-2.0b4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/beust/jcommander/1.27/jcommander-1.27.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/google/guava/guava/14.0.1/guava-14.0.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-apis_2.10/0.17.1-SNAPSHOT/adam-apis_2.10-0.17.1-SNAPSHOT.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/args4j/args4j/2.0.23/args4j-2.0.23.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-cli_2.10/0.17.1-SNAPSHOT/adam-cli_2.10-0.17.1-SNAPSHOT.jar --jars adam-plugins_2.10-0.17.1-SNAPSHOT.jar
--jars adam-plugins_2.10-0.17.1-SNAPSHOT.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-cli/commons-cli/1.2/commons-cli-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-codec/commons-codec/1.4/commons-codec-1.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/slf4j/slf4j-api/1.7.10/slf4j-api-1.7.10.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/log4j/log4j/1.2.17/log4j-1.2.17.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/xerial/snappy/snappy-java/1.1.1.7/snappy-java-1.1.1.7.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/thoughtworks/paranamer/paranamer/2.6/paranamer-2.6.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-io_2.10/0.2.2/utils-io_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-misc_2.10/0.2.2/utils-misc_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/httpcomponents/httpclient/4.3.2/httpclient-4.3.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/httpcomponents/httpcore/4.3.1/httpcore-4.3.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-cli_2.10/0.2.2/utils-cli_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-avro/1.8.1/parquet-avro-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-column/1.8.1/parquet-column-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-common/1.8.1/parquet-common-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-encoding/1.8.1/parquet-encoding-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-hadoop/1.8.1/parquet-hadoop-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-jackson/1.8.1/parquet-jackson-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-format/2.3.0-incubating/parquet-format-2.3.0-incubating.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-metrics_2.10/0.2.2/utils-metrics_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/netflix/servo/servo-core/0.5.5/servo-core-0.5.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/google/code/findbugs/annotations/2.0.0/annotations-2.0.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/scoverage/scalac-scoverage-plugin_2.10/0.99.2/scalac-scoverage-plugin_2.10-0.99.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-io/commons-io/1.3.2/commons-io-1.3.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/bdg-formats/bdg-formats/0.4.0/bdg-formats-0.4.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/avro/avro/1.7.6/avro-1.7.6.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-core_2.10/0.17.1-SNAPSHOT/adam-core_2.10-0.17.1-SNAPSHOT.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/ow2/asm/asm/4.0/asm-4.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/objenesis/objenesis/1.2/objenesis-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/it/unimi/dsi/fastutil/6.4.4/fastutil-6.4.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-scala_2.10/1.8.1/parquet-scala_2.10-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/seqdoop/hadoop-bam/7.0.0/hadoop-bam-7.0.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/github/samtools/htsjdk/1.133/htsjdk-1.133.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/commons/commons-jexl/2.1.1/commons-jexl-2.1.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/tukaani/xz/1.5/xz-1.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/ant/ant/1.8.2/ant-1.8.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/testng/testng/6.8.8/testng-6.8.8.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/beanshell/bsh/2.0b4/bsh-2.0b4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/beust/jcommander/1.27/jcommander-1.27.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/google/guava/guava/14.0.1/guava-14.0.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-apis_2.10/0.17.1-SNAPSHOT/adam-apis_2.10-0.17.1-SNAPSHOT.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/args4j/args4j/2.0.23/args4j-2.0.23.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-cli_2.10/0.17.1-SNAPSHOT/adam-cli_2.10-0.17.1-SNAPSHOT.jar
Member

heuermh commented Aug 5, 2015

I think the single quotes throws it off, this works

$ append_to_option.py , --jars /Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-cli/commons-cli/1.2/commons-cli-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-codec/commons-codec/1.4/commons-codec-1.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/slf4j/slf4j-api/1.7.10/slf4j-api-1.7.10.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/log4j/log4j/1.2.17/log4j-1.2.17.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/xerial/snappy/snappy-java/1.1.1.7/snappy-java-1.1.1.7.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/thoughtworks/paranamer/paranamer/2.6/paranamer-2.6.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-io_2.10/0.2.2/utils-io_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-misc_2.10/0.2.2/utils-misc_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/httpcomponents/httpclient/4.3.2/httpclient-4.3.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/httpcomponents/httpcore/4.3.1/httpcore-4.3.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-cli_2.10/0.2.2/utils-cli_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-avro/1.8.1/parquet-avro-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-column/1.8.1/parquet-column-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-common/1.8.1/parquet-common-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-encoding/1.8.1/parquet-encoding-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-hadoop/1.8.1/parquet-hadoop-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-jackson/1.8.1/parquet-jackson-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-format/2.3.0-incubating/parquet-format-2.3.0-incubating.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-metrics_2.10/0.2.2/utils-metrics_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/netflix/servo/servo-core/0.5.5/servo-core-0.5.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/google/code/findbugs/annotations/2.0.0/annotations-2.0.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/scoverage/scalac-scoverage-plugin_2.10/0.99.2/scalac-scoverage-plugin_2.10-0.99.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-io/commons-io/1.3.2/commons-io-1.3.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/bdg-formats/bdg-formats/0.4.0/bdg-formats-0.4.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/avro/avro/1.7.6/avro-1.7.6.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-core_2.10/0.17.1-SNAPSHOT/adam-core_2.10-0.17.1-SNAPSHOT.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/ow2/asm/asm/4.0/asm-4.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/objenesis/objenesis/1.2/objenesis-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/it/unimi/dsi/fastutil/6.4.4/fastutil-6.4.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-scala_2.10/1.8.1/parquet-scala_2.10-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/seqdoop/hadoop-bam/7.0.0/hadoop-bam-7.0.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/github/samtools/htsjdk/1.133/htsjdk-1.133.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/commons/commons-jexl/2.1.1/commons-jexl-2.1.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/tukaani/xz/1.5/xz-1.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/ant/ant/1.8.2/ant-1.8.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/testng/testng/6.8.8/testng-6.8.8.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/beanshell/bsh/2.0b4/bsh-2.0b4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/beust/jcommander/1.27/jcommander-1.27.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/google/guava/guava/14.0.1/guava-14.0.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-apis_2.10/0.17.1-SNAPSHOT/adam-apis_2.10-0.17.1-SNAPSHOT.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/args4j/args4j/2.0.23/args4j-2.0.23.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-cli_2.10/0.17.1-SNAPSHOT/adam-cli_2.10-0.17.1-SNAPSHOT.jar --jars adam-plugins_2.10-0.17.1-SNAPSHOT.jar
--jars adam-plugins_2.10-0.17.1-SNAPSHOT.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-cli/commons-cli/1.2/commons-cli-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-codec/commons-codec/1.4/commons-codec-1.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/slf4j/slf4j-api/1.7.10/slf4j-api-1.7.10.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/log4j/log4j/1.2.17/log4j-1.2.17.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/xerial/snappy/snappy-java/1.1.1.7/snappy-java-1.1.1.7.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/thoughtworks/paranamer/paranamer/2.6/paranamer-2.6.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-io_2.10/0.2.2/utils-io_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-misc_2.10/0.2.2/utils-misc_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/httpcomponents/httpclient/4.3.2/httpclient-4.3.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/httpcomponents/httpcore/4.3.1/httpcore-4.3.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-cli_2.10/0.2.2/utils-cli_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-avro/1.8.1/parquet-avro-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-column/1.8.1/parquet-column-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-common/1.8.1/parquet-common-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-encoding/1.8.1/parquet-encoding-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-hadoop/1.8.1/parquet-hadoop-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-jackson/1.8.1/parquet-jackson-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-format/2.3.0-incubating/parquet-format-2.3.0-incubating.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/utils/utils-metrics_2.10/0.2.2/utils-metrics_2.10-0.2.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/netflix/servo/servo-core/0.5.5/servo-core-0.5.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/google/code/findbugs/annotations/2.0.0/annotations-2.0.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/scoverage/scalac-scoverage-plugin_2.10/0.99.2/scalac-scoverage-plugin_2.10-0.99.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/commons-io/commons-io/1.3.2/commons-io-1.3.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/bdg-formats/bdg-formats/0.4.0/bdg-formats-0.4.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/avro/avro/1.7.6/avro-1.7.6.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-core_2.10/0.17.1-SNAPSHOT/adam-core_2.10-0.17.1-SNAPSHOT.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/ow2/asm/asm/4.0/asm-4.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/objenesis/objenesis/1.2/objenesis-1.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/it/unimi/dsi/fastutil/6.4.4/fastutil-6.4.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/parquet/parquet-scala_2.10/1.8.1/parquet-scala_2.10-1.8.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/seqdoop/hadoop-bam/7.0.0/hadoop-bam-7.0.0.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/github/samtools/htsjdk/1.133/htsjdk-1.133.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/commons/commons-jexl/2.1.1/commons-jexl-2.1.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/tukaani/xz/1.5/xz-1.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/ant/ant/1.8.2/ant-1.8.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/testng/testng/6.8.8/testng-6.8.8.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/beanshell/bsh/2.0b4/bsh-2.0b4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/beust/jcommander/1.27/jcommander-1.27.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/com/google/guava/guava/14.0.1/guava-14.0.1.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-apis_2.10/0.17.1-SNAPSHOT/adam-apis_2.10-0.17.1-SNAPSHOT.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/args4j/args4j/2.0.23/args4j-2.0.23.jar,/Users/mheuer/working/laserson-adam/adam-cli/target/appassembler/repo/org/bdgenomics/adam/adam-cli_2.10/0.17.1-SNAPSHOT/adam-cli_2.10-0.17.1-SNAPSHOT.jar
@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 5, 2015

Contributor

Yeah, I had to remove the " on a few variables bc of the single quotes, which wasn't a problem before. Man I hate bash...

ok, pushed a commit that should fix that i think

Contributor

laserson commented Aug 5, 2015

Yeah, I had to remove the " on a few variables bc of the single quotes, which wasn't a problem before. Man I hate bash...

ok, pushed a commit that should fix that i think

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 5, 2015

Member

Thanks, works for me.

Member

heuermh commented Aug 5, 2015

Thanks, works for me.

Show outdated Hide outdated README.md
@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 5, 2015

Contributor

Ok, hopefully done now. Now, if the -- is missing, it'll assume all options are ADAM options. I reverted the relevant changes in the docs etc.

Contributor

laserson commented Aug 5, 2015

Ok, hopefully done now. Now, if the -- is missing, it'll assume all options are ADAM options. I reverted the relevant changes in the docs etc.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 5, 2015

Member

+1

I've been testing this for a while, and haven't run into any problems yet.

Member

heuermh commented Aug 5, 2015

+1

I've been testing this for a while, and haven't run into any problems yet.

@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 5, 2015

Contributor

Ok, adam-submit should not default to PATH spark-submit. This was already the case for adam-shell and adam-pyspark.

Contributor

laserson commented Aug 5, 2015

Ok, adam-submit should not default to PATH spark-submit. This was already the case for adam-shell and adam-pyspark.

@ryan-williams

This comment has been minimized.

Show comment
Hide comment
@ryan-williams

ryan-williams Aug 5, 2015

worth erroring if both these branches fail to find a spark-shell? Or do we error in that case somewhere upstream?

ryan-williams commented on bin/adam-shell in aa55354 Aug 5, 2015

worth erroring if both these branches fail to find a spark-shell? Or do we error in that case somewhere upstream?

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 5, 2015

Owner
Owner

laserson replied Aug 5, 2015

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 5, 2015

Member

Any idea why Jenkins is choking on these? Doesn't look like a code problem

> git checkout -f e0e5a8bad010d001f76374e96da36d90ca7c2533
FATAL: Could not checkout null with start point e0e5a8bad010d001f76374e96da36d90ca7c2533
hudson.plugins.git.GitException: Could not checkout null with start point e0e5a8bad010d001f76374e96da36d90ca7c2533
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1855)
...

https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/818/SCALAVER=2.10,hadoop.version=2.2.0,label=centos/console

Member

heuermh commented Aug 5, 2015

Any idea why Jenkins is choking on these? Doesn't look like a code problem

> git checkout -f e0e5a8bad010d001f76374e96da36d90ca7c2533
FATAL: Could not checkout null with start point e0e5a8bad010d001f76374e96da36d90ca7c2533
hudson.plugins.git.GitException: Could not checkout null with start point e0e5a8bad010d001f76374e96da36d90ca7c2533
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1855)
...

https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/818/SCALAVER=2.10,hadoop.version=2.2.0,label=centos/console

@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 5, 2015

Contributor

Jenkins, retest this please.

Contributor

laserson commented Aug 5, 2015

Jenkins, retest this please.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Aug 6, 2015

Member

Is this ready for merge, or are there still pending issues?

Member

fnothaft commented Aug 6, 2015

Is this ready for merge, or are there still pending issues?

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 6, 2015

Member

+1 from me above and now Jenkins is happy again. Anyone else?

Member

heuermh commented Aug 6, 2015

+1 from me above and now Jenkins is happy again. Anyone else?

@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 6, 2015

Contributor

+1 from my end

Contributor

laserson commented Aug 6, 2015

+1 from my end

massie added a commit that referenced this pull request Aug 6, 2015

@massie massie merged commit ad277c3 into bigdatagenomics:master Aug 6, 2015

1 check passed

default Merged build finished.
Details
@massie

This comment has been minimized.

Show comment
Hide comment
@massie

massie Aug 6, 2015

Member

Thanks, Uri!

Member

massie commented Aug 6, 2015

Thanks, Uri!

@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 6, 2015

Contributor

argh, sorry, i neglected to squash

Contributor

laserson commented Aug 6, 2015

argh, sorry, i neglected to squash

@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Aug 6, 2015

Contributor

you have a problem with me squashing and force-pushing to upstream?

Contributor

laserson commented Aug 6, 2015

you have a problem with me squashing and force-pushing to upstream?

@massie

This comment has been minimized.

Show comment
Hide comment
@massie

massie Aug 6, 2015

Member

Let's not do that. It's okay if everyone knows you hate bash. :)

Member

massie commented Aug 6, 2015

Let's not do that. It's okay if everyone knows you hate bash. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment