Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dependency and plugin versions #1360

Merged
merged 1 commit into from Apr 6, 2017

Conversation

@heuermh
Copy link
Member

@heuermh heuermh commented Jan 20, 2017

Notes to self:

@fnothaft
Copy link
Member

@fnothaft fnothaft commented Jan 20, 2017

Nice! We should probably hold on going to Avro 1.8.2 until they have an official release and not just a RC, no? Otherwise, SGTM. Let me look into that servo dependency...

@heuermh
Copy link
Member Author

@heuermh heuermh commented Jan 20, 2017

Yeah, I'm still investigating this and will have further commits. I just wanted to have Jenkins confirm things piecemeal for me, cuz running the Jenkins tests on my laptop takes forever.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jan 20, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1745/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 2b5738672a75951f9403dc9e8bdbef143fc96e71 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 2b5738672a75951f9403dc9e8bdbef143fc96e71 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 2b5738672a75951f9403dc9e8bdbef143fc96e71First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Jan 20, 2017

bumping servo from 0.5.5 to 0.10.0 or 0.12.12 upstream in utils results in compile errors

[INFO] Compiling 22 source files to /Users/heuermh/working/bdg-utils/utils-metrics/target/2.10.6/classes at 1484931081411
[WARNING] warning: Class org.slf4j.Logger not found - continuing with a stub.
[ERROR] /Users/heuermh/working/bdg-utils/utils-metrics/src/main/scala/org/bdgenomics/utils/instrumentation/Metrics.scala:323: error: class StringMonitor needs to be abstract, since method getValue in trait Monitor of type (x$1: Int)String is not defined
[ERROR]   private class StringMonitor(name: String, value: String, tags: Tag*) extends Monitor[String] {
[ERROR]                 ^
[ERROR] /Users/heuermh/working/bdg-utils/utils-metrics/src/main/scala/org/bdgenomics/utils/instrumentation/ServoTimer.scala:44: error: class ServoTimer needs to be abstract, since method getValue in trait Monitor of type (x$1: Int)Object is not defined
[ERROR] class ServoTimer(name: String, @transient tags: Tag*) extends ConfigurableMonitor(name, tags)
[ERROR]       ^
[ERROR] /Users/heuermh/working/bdg-utils/utils-metrics/src/main/scala/org/bdgenomics/utils/instrumentation/ServoTimer.scala:161: error: class LongValueStatMonitor needs to be abstract, since method getValue in trait Monitor of type (x$1: Int)Long is not defined
[ERROR]   private class LongValueStatMonitor(name: String, @transient tags: Seq[Tag], function: () => Long)
[ERROR]                 ^
[ERROR] /Users/heuermh/working/bdg-utils/utils-metrics/src/main/scala/org/bdgenomics/utils/instrumentation/ServoTimer.scala:168: error: class ConditionalGauge needs to be abstract, since method getValue in trait Monitor of type (x$1: Int)Long is not defined
[ERROR]   private class ConditionalGauge(name: String, @transient tags: Seq[Tag], condition: (Long, Long) => Boolean)
[ERROR]                 ^
[WARNING] one warning found
[ERROR] four errors found
@heuermh
Copy link
Member Author

@heuermh heuermh commented Jan 20, 2017

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jan 20, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1746/
Test PASSed.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jan 20, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1747/
Test PASSed.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jan 23, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1749/
Test PASSed.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jan 23, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1751/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 1722b2f694cda28eca8fbee12830993b79e738ca # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 1722b2f694cda28eca8fbee12830993b79e738ca (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 1722b2f694cda28eca8fbee12830993b79e738caFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Jan 23, 2017

I also see the exception in the Jenkins build failure logs when running adam-submit locally.

java.lang.NoSuchMethodError: org.apache.avro.Schema.getLogicalType()Lorg/apache/avro/LogicalType;
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:178)
	at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:214)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:171)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:130)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:227)
	at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:124)
	at org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:115)
	at org.apache.parquet.avro.AvroWriteSupport.init(AvroWriteSupport.java:117)
	at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:288)
	at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:262)
	at org.apache.spark.rdd.InstrumentedOutputFormat.getRecordWriter(InstrumentedOutputFormat.scala:35)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:88)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I'm not sure where this runtime conflict is coming from. The method Schema.getLogicalType() was added to avro post version 1.7.7, and all our dependencies should be at least avro version 1.8.0.

$ mvn dependency:tree | grep avro
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.7.7:provided
[INFO] |  |  \- org.apache.avro:avro-ipc:jar:tests:1.7.7:provided
[INFO] +- org.apache.avro:avro:jar:1.8.1:compile
[INFO] +- org.apache.parquet:parquet-avro:jar:1.8.2:compile

Oops.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jan 23, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1753/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 0771250a0a40914ca60b6829744bb28ab1a3dafe # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 0771250a0a40914ca60b6829744bb28ab1a3dafe (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 0771250a0a40914ca60b6829744bb28ab1a3dafeFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Jan 23, 2017

Jenkins, retest this please

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jan 23, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1754/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 3c907d9 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 3c907d9 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 3c907d948e99a2802a4692e82dc3936e947d3f57First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Jan 24, 2017

Jenkins, retest this please

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jan 24, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1757/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 3c907d9 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 3c907d9 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 3c907d948e99a2802a4692e82dc3936e947d3f57First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Jan 24, 2017

Still not sure what I'm missing here, all the dependency versions are 1.8.1 yet something is running into a 1.7.7 or earlier version of the avro Schema class at runtime

$ mvn dependency:tree | grep avro
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.1:provided
[INFO] |  |  \- org.apache.avro:avro-ipc:jar:1.8.1:provided
[INFO] +- org.apache.avro:avro:jar:1.8.1:compile
[INFO] +- org.apache.parquet:parquet-avro:jar:1.8.2:compile
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.1:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.8.1:provided
[INFO] |  \- org.apache.avro:avro:jar:1.8.1:compile
[INFO] |  +- org.apache.parquet:parquet-avro:jar:1.8.2:compile
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.1:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.8.1:provided
[INFO] |  \- org.apache.parquet:parquet-avro:jar:1.8.2:compile
[INFO] |  \- org.apache.avro:avro:jar:1.8.1:compile
[INFO] |  |  \- org.apache.parquet:parquet-avro:jar:1.8.2:compile
[INFO] |  |  \- org.apache.avro:avro:jar:1.8.1:compile

(duplicates here because I am running from the parent directory)

Avro 1.7.7 is present in the Spark lib folder

$ ls /usr/local/Cellar/apache-spark/2.1.0/libexec/jars/avro-*
/usr/local/Cellar/apache-spark/2.1.0/libexec/jars/avro-1.7.7.jar
/usr/local/Cellar/apache-spark/2.1.0/libexec/jars/avro-ipc-1.7.7.jar
/usr/local/Cellar/apache-spark/2.1.0/libexec/jars/avro-mapred-1.7.7-hadoop2.jar

Are we not able to fully override these at runtime?

@fnothaft
Copy link
Member

@fnothaft fnothaft commented Jan 24, 2017

Sigh! How does this work right now? I show us on Avro 1.8.0 on ToT?

@heuermh
Copy link
Member Author

@heuermh heuermh commented Jan 24, 2017

No. As the parquet 1.8.2 release is still being voted on, we have the chance to -1 if it is at fault.

@heuermh heuermh force-pushed the heuermh:dep-versions branch from 55de54f to 93ebafb Feb 21, 2017
@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Feb 21, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1810/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains b463c5e # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision b463c5e (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b463c5e5d0b076f54a213ddf3b59d2532afd61f7First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Feb 21, 2017

Am I seeing this right, are only the Scala 2.10 builds failing now?

Whenever I run scripts/jenkins-test locally, it fails with

+ echo 'Applying move_to_xyz script marred a pom.xml file.'
Applying move_to_xyz script marred a pom.xml file.
+ echo Exiting...
Exiting...
+ exit 1
@fnothaft
Copy link
Member

@fnothaft fnothaft commented Feb 21, 2017

Yes, although your problems are worse than you imagined. See, only the Scala 2.10 builds run the integration tests. Check out your stacktrace from the integration tests:

2017-02-21 13:27:20 ERROR Executor:95 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoSuchMethodError: org.apache.avro.Schema.getLogicalType()Lorg/apache/avro/LogicalType;
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:178)
	at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:214)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:171)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:130)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:227)
	at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:124)
	at org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:115)
	at org.apache.parquet.avro.AvroWriteSupport.init(AvroWriteSupport.java:117)
	at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:288)
	at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:262)
	at org.apache.spark.rdd.InstrumentedOutputFormat.getRecordWriter(InstrumentedOutputFormat.scala:35)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1107)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1091)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Looks like a Parquet/Avro version incompatibility.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Feb 21, 2017

That is the problem I'm trying to solve. You asked earlier

How does this work right now?

I don't know how it works right now, nor do I understand where the < 1.8.0 version of avro is coming into play.

@fnothaft
Copy link
Member

@fnothaft fnothaft commented Feb 21, 2017

Oh hah, right. Yeah, let me circle back and look at this.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Feb 22, 2017

Reported as Apache Spark JIRA issue
https://issues.apache.org/jira/browse/SPARK-19697

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Feb 22, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1814/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d603da6 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision d603da6 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d603da6ff3078c433ac94bac119a3a456156abdaFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@coveralls
Copy link

@coveralls coveralls commented Mar 3, 2017

Coverage Status

Changes Unknown when pulling 9f10a85 on heuermh:dep-versions into ** on bigdatagenomics:master**.

1 similar comment
@coveralls
Copy link

@coveralls coveralls commented Mar 3, 2017

Coverage Status

Changes Unknown when pulling 9f10a85 on heuermh:dep-versions into ** on bigdatagenomics:master**.

@heuermh heuermh force-pushed the heuermh:dep-versions branch from bd510c8 to 812816b Mar 14, 2017
@coveralls
Copy link

@coveralls coveralls commented Mar 14, 2017

Coverage Status

Coverage remained the same at 76.659% when pulling 812816b on heuermh:dep-versions into b8477dc on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Mar 14, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1875/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 3b93d21 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 3b93d21 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 3b93d21628948f35070a7713fe2cc8f8e81ec537First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Mar 15, 2017

I am not seeing these unit test failures locally. Any ideas? I'll try scripts/jenkins-test locally next.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Mar 15, 2017

Not sure what the coveralls token should be...

$ HADOOP_VERSION=2.6.0 SPARK_VERSION=1.6.1 ./scripts/jenkins-test
...
# if this is a pull request, we need to set the coveralls pr id
if [[ ! -z $ghprbPullId ]];
then
    COVERALLS_PRB_OPTION="-DpullRequest=${ghprbPullId}"
fi
+ [[ ! -z '' ]]

# coveralls token should not be visible
set +x +v
+ set +x +v
Coveralls token is not set. Exiting...
@fnothaft
Copy link
Member

@fnothaft fnothaft commented Mar 15, 2017

@heuermh can you back out the upgrade to utils 0.2.12? I'm wondering if we scoobied something in the IntervalTree code in the 0.2.12 release.

@heuermh heuermh force-pushed the heuermh:dep-versions branch from 812816b to 40f774c Mar 16, 2017
@coveralls
Copy link

@coveralls coveralls commented Mar 16, 2017

Coverage Status

Coverage remained the same at 76.479% when pulling 40f774c on heuermh:dep-versions into cf39e6c on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Mar 16, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1882/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains bbc8a00 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision bbc8a00 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f bbc8a00d637fafa01e88312da46ebfd2f49747dbFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Mar 16, 2017

I still can't reproduce these failures locally, and the failures on Jenkins are not consistent.

@devin-petersohn
Copy link
Member

@devin-petersohn devin-petersohn commented Mar 16, 2017

I found the issue. It revolves around our thread unsafe variable that was meant to speed up the queries in IntervalArray. I am having the same issue in #1324. I think we should revert that change immediately. I am working on a PR for utils. Here are the lines in question: Thread unsafe variable declaration, Use in the get method. Once I took this out I no longer got the failures.

@coveralls
Copy link

@coveralls coveralls commented Apr 6, 2017

Coverage Status

Coverage remained the same at 81.535% when pulling 9abb159 on heuermh:dep-versions into 98d2c92 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Apr 6, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1933/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1360/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 25a0a3fec82893638593d8d849a6cee0d6a99086 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1360/merge^{commit} # timeout=10Checking out Revision 25a0a3fec82893638593d8d849a6cee0d6a99086 (origin/pr/1360/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 25a0a3fec82893638593d8d849a6cee0d6a99086First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh heuermh force-pushed the heuermh:dep-versions branch from 9abb159 to 98ba92e Apr 6, 2017
@heuermh
Copy link
Member Author

@heuermh heuermh commented Apr 6, 2017

Fixes #1300

@coveralls
Copy link

@coveralls coveralls commented Apr 6, 2017

Coverage Status

Coverage remained the same at 81.535% when pulling 98ba92e on heuermh:dep-versions into 98d2c92 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Apr 6, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1934/
Test PASSed.

@fnothaft
Copy link
Member

@fnothaft fnothaft commented Apr 6, 2017

@heuermh this is good to go now, correct?

@heuermh
Copy link
Member Author

@heuermh heuermh commented Apr 6, 2017

Yes, I believe so

@fnothaft fnothaft merged commit 9505d47 into bigdatagenomics:master Apr 6, 2017
3 checks passed
3 checks passed
codacy/pr Good work! A positive pull request.
Details
coverage/coveralls Coverage remained the same at 81.535%
Details
default Merged build finished.
Details
@fnothaft
Copy link
Member

@fnothaft fnothaft commented Apr 6, 2017

Merged! Thanks @heuermh!

@heuermh heuermh deleted the heuermh:dep-versions branch Apr 6, 2017
@@ -93,14 +93,17 @@
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<scope>compile</scope>

This comment has been minimized.

@ryan-williams

ryan-williams Apr 6, 2017
Member

why add all these, OOC?

This comment has been minimized.

@heuermh

heuermh Apr 6, 2017
Author Member

for doc mostly, scope is compile by default; these make it explicit

@heuermh heuermh moved this from Triage to Completed in Release 0.23.0 May 30, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.