Update dependency and plugin versions #1360

heuermh · 2017-01-20T16:30:10Z

Notes to self:

HadoopGenomics/Hadoop-BAM@7.7.0...7.7.1
apache/avro@release-1.8.0...release-1.8.1
apache/avro@release-1.8.1...release-1.8.2-rc1
qos-ch/slf4j@v_1.7.21...v_1.7.22
slf4j-log4j12 is compile scoped in adam-core, that doesn't seem correct
args4j version 2.32 is not binary compatible with 2.0.31, fix upstream in utils, may require code changes
scoverage 1.1.1 to 1.3.0, see issue Scoverage plugin not used properly utils#95
servo dependency was reported unused; older version declared in utils-metrics, may be req'd for guava exclusion
update htsjdk 2.5.0 to 2.8.1 in separate commit, to isolate any issues
guava is still stuck by hadoop as far as I know, follow up
parquet 1.9.0 is available but not used in spark, spark dev mailing list was discussing a 1.8.2 release, follow up

fnothaft · 2017-01-20T16:34:22Z

Nice! We should probably hold on going to Avro 1.8.2 until they have an official release and not just a RC, no? Otherwise, SGTM. Let me look into that servo dependency...

heuermh · 2017-01-20T16:37:06Z

Yeah, I'm still investigating this and will have further commits. I just wanted to have Jenkins confirm things piecemeal for me, cuz running the Jenkins tests on my laptop takes forever.

AmplabJenkins · 2017-01-20T16:44:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1745/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 2b5738672a75951f9403dc9e8bdbef143fc96e71 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 2b5738672a75951f9403dc9e8bdbef143fc96e71 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 2b5738672a75951f9403dc9e8bdbef143fc96e71First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-01-20T16:54:10Z

bumping servo from 0.5.5 to 0.10.0 or 0.12.12 upstream in utils results in compile errors

[INFO] Compiling 22 source files to /Users/heuermh/working/bdg-utils/utils-metrics/target/2.10.6/classes at 1484931081411
[WARNING] warning: Class org.slf4j.Logger not found - continuing with a stub.
[ERROR] /Users/heuermh/working/bdg-utils/utils-metrics/src/main/scala/org/bdgenomics/utils/instrumentation/Metrics.scala:323: error: class StringMonitor needs to be abstract, since method getValue in trait Monitor of type (x$1: Int)String is not defined
[ERROR]   private class StringMonitor(name: String, value: String, tags: Tag*) extends Monitor[String] {
[ERROR]                 ^
[ERROR] /Users/heuermh/working/bdg-utils/utils-metrics/src/main/scala/org/bdgenomics/utils/instrumentation/ServoTimer.scala:44: error: class ServoTimer needs to be abstract, since method getValue in trait Monitor of type (x$1: Int)Object is not defined
[ERROR] class ServoTimer(name: String, @transient tags: Tag*) extends ConfigurableMonitor(name, tags)
[ERROR]       ^
[ERROR] /Users/heuermh/working/bdg-utils/utils-metrics/src/main/scala/org/bdgenomics/utils/instrumentation/ServoTimer.scala:161: error: class LongValueStatMonitor needs to be abstract, since method getValue in trait Monitor of type (x$1: Int)Long is not defined
[ERROR]   private class LongValueStatMonitor(name: String, @transient tags: Seq[Tag], function: () => Long)
[ERROR]                 ^
[ERROR] /Users/heuermh/working/bdg-utils/utils-metrics/src/main/scala/org/bdgenomics/utils/instrumentation/ServoTimer.scala:168: error: class ConditionalGauge needs to be abstract, since method getValue in trait Monitor of type (x$1: Int)Long is not defined
[ERROR]   private class ConditionalGauge(name: String, @transient tags: Seq[Tag], condition: (Long, Long) => Boolean)
[ERROR]                 ^
[WARNING] one warning found
[ERROR] four errors found

heuermh · 2017-01-20T17:01:35Z

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 1722b2f694cda28eca8fbee12830993b79e738ca # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 1722b2f694cda28eca8fbee12830993b79e738ca (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 1722b2f694cda28eca8fbee12830993b79e738caFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-01-23T21:46:44Z

I also see the exception in the Jenkins build failure logs when running adam-submit locally.

java.lang.NoSuchMethodError: org.apache.avro.Schema.getLogicalType()Lorg/apache/avro/LogicalType;
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:178)
	at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:214)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:171)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:130)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:227)
	at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:124)
	at org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:115)
	at org.apache.parquet.avro.AvroWriteSupport.init(AvroWriteSupport.java:117)
	at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:288)
	at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:262)
	at org.apache.spark.rdd.InstrumentedOutputFormat.getRecordWriter(InstrumentedOutputFormat.scala:35)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:88)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I'm not sure where this runtime conflict is coming from. The method Schema.getLogicalType() was added to avro post version 1.7.7, and all our dependencies should be at least avro version 1.8.0.

$ mvn dependency:tree | grep avro
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.7.7:provided
[INFO] |  |  \- org.apache.avro:avro-ipc:jar:tests:1.7.7:provided
[INFO] +- org.apache.avro:avro:jar:1.8.1:compile
[INFO] +- org.apache.parquet:parquet-avro:jar:1.8.2:compile

Oops.

AmplabJenkins · 2017-01-23T22:33:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1753/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 0771250a0a40914ca60b6829744bb28ab1a3dafe # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 0771250a0a40914ca60b6829744bb28ab1a3dafe (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 0771250a0a40914ca60b6829744bb28ab1a3dafeFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-01-23T22:35:37Z

Jenkins, retest this please

AmplabJenkins · 2017-01-23T22:48:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1754/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 3c907d9 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 3c907d9 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 3c907d948e99a2802a4692e82dc3936e947d3f57First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-01-24T00:30:56Z

Jenkins, retest this please

AmplabJenkins · 2017-01-24T00:42:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1757/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 3c907d9 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 3c907d9 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 3c907d948e99a2802a4692e82dc3936e947d3f57First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-01-24T02:17:28Z

Still not sure what I'm missing here, all the dependency versions are 1.8.1 yet something is running into a 1.7.7 or earlier version of the avro Schema class at runtime

$ mvn dependency:tree | grep avro
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.1:provided
[INFO] |  |  \- org.apache.avro:avro-ipc:jar:1.8.1:provided
[INFO] +- org.apache.avro:avro:jar:1.8.1:compile
[INFO] +- org.apache.parquet:parquet-avro:jar:1.8.2:compile
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.1:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.8.1:provided
[INFO] |  \- org.apache.avro:avro:jar:1.8.1:compile
[INFO] |  +- org.apache.parquet:parquet-avro:jar:1.8.2:compile
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.1:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.8.1:provided
[INFO] |  \- org.apache.parquet:parquet-avro:jar:1.8.2:compile
[INFO] |  \- org.apache.avro:avro:jar:1.8.1:compile
[INFO] |  |  \- org.apache.parquet:parquet-avro:jar:1.8.2:compile
[INFO] |  |  \- org.apache.avro:avro:jar:1.8.1:compile

(duplicates here because I am running from the parent directory)

Avro 1.7.7 is present in the Spark lib folder

$ ls /usr/local/Cellar/apache-spark/2.1.0/libexec/jars/avro-*
/usr/local/Cellar/apache-spark/2.1.0/libexec/jars/avro-1.7.7.jar
/usr/local/Cellar/apache-spark/2.1.0/libexec/jars/avro-ipc-1.7.7.jar
/usr/local/Cellar/apache-spark/2.1.0/libexec/jars/avro-mapred-1.7.7-hadoop2.jar

Are we not able to fully override these at runtime?

fnothaft · 2017-01-24T05:16:39Z

Sigh! How does this work right now? I show us on Avro 1.8.0 on ToT?

heuermh · 2017-01-24T13:10:47Z

No. As the parquet 1.8.2 release is still being voted on, we have the chance to -1 if it is at fault.

AmplabJenkins · 2017-02-21T21:28:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1810/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains b463c5e # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision b463c5e (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b463c5e5d0b076f54a213ddf3b59d2532afd61f7First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-02-21T22:35:01Z

Am I seeing this right, are only the Scala 2.10 builds failing now?

Whenever I run scripts/jenkins-test locally, it fails with

+ echo 'Applying move_to_xyz script marred a pom.xml file.'
Applying move_to_xyz script marred a pom.xml file.
+ echo Exiting...
Exiting...
+ exit 1

fnothaft · 2017-02-21T22:56:25Z

Yes, although your problems are worse than you imagined. See, only the Scala 2.10 builds run the integration tests. Check out your stacktrace from the integration tests:

2017-02-21 13:27:20 ERROR Executor:95 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoSuchMethodError: org.apache.avro.Schema.getLogicalType()Lorg/apache/avro/LogicalType;
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:178)
	at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:214)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:171)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:130)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:227)
	at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:124)
	at org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:115)
	at org.apache.parquet.avro.AvroWriteSupport.init(AvroWriteSupport.java:117)
	at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:288)
	at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:262)
	at org.apache.spark.rdd.InstrumentedOutputFormat.getRecordWriter(InstrumentedOutputFormat.scala:35)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1107)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1091)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Looks like a Parquet/Avro version incompatibility.

heuermh · 2017-02-21T23:05:38Z

That is the problem I'm trying to solve. You asked earlier

How does this work right now?

I don't know how it works right now, nor do I understand where the < 1.8.0 version of avro is coming into play.

fnothaft · 2017-02-21T23:32:42Z

Oh hah, right. Yeah, let me circle back and look at this.

heuermh · 2017-02-22T17:29:10Z

Reported as Apache Spark JIRA issue
https://issues.apache.org/jira/browse/SPARK-19697

AmplabJenkins · 2017-02-22T17:42:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1814/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d603da6 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision d603da6 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d603da6ff3078c433ac94bac119a3a456156abdaFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

coveralls · 2017-03-03T23:15:57Z

Changes Unknown when pulling 9f10a85 on heuermh:dep-versions into ** on bigdatagenomics:master**.

coveralls · 2017-03-03T23:15:57Z

Changes Unknown when pulling 9f10a85 on heuermh:dep-versions into ** on bigdatagenomics:master**.

fnothaft · 2017-03-14T22:51:15Z

Also, any chance I can get you to rebase this?

heuermh · 2017-03-14T22:52:27Z

Yep, looks like a rebase should help. Unit tests work fine for me locally.

coveralls · 2017-03-14T23:21:40Z

Coverage remained the same at 76.659% when pulling 812816b on heuermh:dep-versions into b8477dc on bigdatagenomics:master.

AmplabJenkins · 2017-03-14T23:25:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1875/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 3b93d21 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 3b93d21 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 3b93d21628948f35070a7713fe2cc8f8e81ec537First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-03-15T00:41:29Z

I am not seeing these unit test failures locally. Any ideas? I'll try scripts/jenkins-test locally next.

heuermh · 2017-03-15T00:44:00Z

Not sure what the coveralls token should be...

$ HADOOP_VERSION=2.6.0 SPARK_VERSION=1.6.1 ./scripts/jenkins-test
...
# if this is a pull request, we need to set the coveralls pr id
if [[ ! -z $ghprbPullId ]];
then
    COVERALLS_PRB_OPTION="-DpullRequest=${ghprbPullId}"
fi
+ [[ ! -z '' ]]

# coveralls token should not be visible
set +x +v
+ set +x +v
Coveralls token is not set. Exiting...

fnothaft · 2017-03-15T05:37:35Z

@heuermh can you back out the upgrade to utils 0.2.12? I'm wondering if we scoobied something in the IntervalTree code in the 0.2.12 release.

coveralls · 2017-03-16T15:10:29Z

Coverage remained the same at 76.479% when pulling 40f774c on heuermh:dep-versions into cf39e6c on bigdatagenomics:master.

AmplabJenkins · 2017-03-16T15:13:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1882/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains bbc8a00 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision bbc8a00 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f bbc8a00d637fafa01e88312da46ebfd2f49747dbFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-03-16T16:18:53Z

I still can't reproduce these failures locally, and the failures on Jenkins are not consistent.

devin-petersohn · 2017-03-16T19:26:51Z

I found the issue. It revolves around our thread unsafe variable that was meant to speed up the queries in IntervalArray. I am having the same issue in #1324. I think we should revert that change immediately. I am working on a PR for utils. Here are the lines in question: Thread unsafe variable declaration, Use in the get method. Once I took this out I no longer got the failures.

coveralls · 2017-04-06T02:46:04Z

Coverage remained the same at 81.535% when pulling 9abb159 on heuermh:dep-versions into 98d2c92 on bigdatagenomics:master.

AmplabJenkins · 2017-04-06T02:50:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1933/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 25a0a3fec82893638593d8d849a6cee0d6a99086 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 25a0a3fec82893638593d8d849a6cee0d6a99086 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 25a0a3fec82893638593d8d849a6cee0d6a99086First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-04-06T02:59:28Z

Fixes #1300

coveralls · 2017-04-06T03:06:34Z

Coverage remained the same at 81.535% when pulling 98ba92e on heuermh:dep-versions into 98d2c92 on bigdatagenomics:master.

AmplabJenkins · 2017-04-06T03:10:22Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1934/
Test PASSed.

fnothaft · 2017-04-06T15:35:53Z

@heuermh this is good to go now, correct?

heuermh · 2017-04-06T15:51:20Z

Yes, I believe so

fnothaft · 2017-04-06T16:22:41Z

Merged! Thanks @heuermh!

ryan-williams · 2017-04-06T18:05:38Z

adam-apis/pom.xml

@@ -93,14 +93,17 @@
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
+      <scope>compile</scope>


why add all these, OOC?

for doc mostly, scope is compile by default; these make it explicit

heuermh force-pushed the dep-versions branch from 55de54f to 93ebafb Compare February 21, 2017 21:16

heuermh mentioned this pull request Feb 22, 2017

Add Apache Spark version to --version text #1400

Merged

heuermh mentioned this pull request Mar 3, 2017

Update dependency and plugin versions bigdatagenomics/utils#101

Merged

heuermh force-pushed the dep-versions branch from bd510c8 to 812816b Compare March 14, 2017 23:00

fnothaft mentioned this pull request Mar 16, 2017

Efficient Joins and (re)Partitioning #1324

Merged

heuermh force-pushed the dep-versions branch from 812816b to 40f774c Compare March 16, 2017 15:01

fnothaft mentioned this pull request Mar 16, 2017

Dependency version bump + BroadcastRegionJoin fix #1440

Merged

heuermh mentioned this pull request Mar 17, 2017

[UTILS-114] Add abstract duplicate method to IntervalArray. bigdatagenomics/utils#116

Merged

heuermh force-pushed the dep-versions branch from 40f774c to 9abb159 Compare April 6, 2017 02:36

Update dependency and plugin versions

98ba92e

heuermh force-pushed the dep-versions branch from 9abb159 to 98ba92e Compare April 6, 2017 02:56

fnothaft merged commit 9505d47 into bigdatagenomics:master Apr 6, 2017

heuermh deleted the dep-versions branch April 6, 2017 16:58

ryan-williams reviewed Apr 6, 2017

View reviewed changes

Update dependency and plugin versions #1360

Update dependency and plugin versions #1360

Conversation

heuermh commented Jan 20, 2017 • edited Loading

fnothaft commented Jan 20, 2017

heuermh commented Jan 20, 2017

AmplabJenkins commented Jan 20, 2017

Build result: FAILURE

heuermh commented Jan 20, 2017

heuermh commented Jan 20, 2017

AmplabJenkins commented Jan 20, 2017

AmplabJenkins commented Jan 20, 2017

AmplabJenkins commented Jan 23, 2017

AmplabJenkins commented Jan 23, 2017

Build result: FAILURE

heuermh commented Jan 23, 2017 • edited Loading

AmplabJenkins commented Jan 23, 2017

Build result: FAILURE

heuermh commented Jan 23, 2017

AmplabJenkins commented Jan 23, 2017

Build result: FAILURE

heuermh commented Jan 24, 2017

AmplabJenkins commented Jan 24, 2017

Build result: FAILURE

heuermh commented Jan 24, 2017 • edited Loading

fnothaft commented Jan 24, 2017

heuermh commented Jan 24, 2017

AmplabJenkins commented Feb 21, 2017

Build result: FAILURE

heuermh commented Feb 21, 2017

fnothaft commented Feb 21, 2017

heuermh commented Feb 21, 2017

fnothaft commented Feb 21, 2017

heuermh commented Feb 22, 2017

AmplabJenkins commented Feb 22, 2017

Build result: FAILURE

coveralls commented Mar 3, 2017

coveralls commented Mar 3, 2017 • edited Loading

fnothaft commented Mar 14, 2017

heuermh commented Mar 14, 2017 • edited Loading

coveralls commented Mar 14, 2017 • edited Loading

AmplabJenkins commented Mar 14, 2017

Build result: FAILURE

heuermh commented Mar 15, 2017 • edited Loading

heuermh commented Mar 15, 2017

fnothaft commented Mar 15, 2017

coveralls commented Mar 16, 2017 • edited Loading

AmplabJenkins commented Mar 16, 2017

Build result: FAILURE

heuermh commented Mar 16, 2017

devin-petersohn commented Mar 16, 2017 • edited Loading

coveralls commented Apr 6, 2017 • edited Loading

AmplabJenkins commented Apr 6, 2017

Build result: FAILURE

heuermh commented Apr 6, 2017

coveralls commented Apr 6, 2017 • edited Loading

AmplabJenkins commented Apr 6, 2017

fnothaft commented Apr 6, 2017

heuermh commented Apr 6, 2017

fnothaft commented Apr 6, 2017

ryan-williams Apr 6, 2017

Choose a reason for hiding this comment

heuermh Apr 6, 2017

Choose a reason for hiding this comment

heuermh commented Jan 20, 2017 •

edited

Loading

heuermh commented Jan 23, 2017 •

edited

Loading

heuermh commented Jan 24, 2017 •

edited

Loading

coveralls commented Mar 3, 2017 •

edited

Loading

heuermh commented Mar 14, 2017 •

edited

Loading

coveralls commented Mar 14, 2017 •

edited

Loading

heuermh commented Mar 15, 2017 •

edited

Loading

coveralls commented Mar 16, 2017 •

edited

Loading

devin-petersohn commented Mar 16, 2017 •

edited

Loading

coveralls commented Apr 6, 2017 •

edited

Loading

coveralls commented Apr 6, 2017 •

edited

Loading