New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CANNOLI-94][CANNOLI-95] Drop support for Scala 2.10 and Spark 1.x. #100

Closed
wants to merge 2 commits into
base: master
from

Conversation

Projects
None yet
2 participants
@heuermh
Member

heuermh commented Feb 5, 2018

Fixes #94.
Fixes #95.

Work in progress, currently hangs unit tests at

- interleave two paired FASTQ files
Exception in thread "dag-scheduler-event-loop" java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.InputSplit.getLocationInfo()[Lorg/apache/hadoop/mapred/SplitLocationInfo;
	at org.apache.spark.rdd.NewHadoopRDD.getPreferredLocations(NewHadoopRDD.scala:282)
	at org.apache.spark.rdd.RDD$$anonfun$preferredLocations$2.apply(RDD.scala:274)
	at org.apache.spark.rdd.RDD$$anonfun$preferredLocations$2.apply(RDD.scala:274)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.rdd.RDD.preferredLocations(RDD.scala:273)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:1633)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:1644)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:1643)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:1643)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:1643)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:1641)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:1641)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:1644)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:1643)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:1643)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:1643)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:1641)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:1641)
	at org.apache.spark.scheduler.DAGScheduler.getPreferredLocs(DAGScheduler.scala:1607)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$16.apply(DAGScheduler.scala:973)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$16.apply(DAGScheduler.scala:971)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:971)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:930)
	at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:874)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1695)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1687)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1676)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

Build targets in Jenkins will need to be updated.

@heuermh heuermh added this to the 0.2.0 milestone Feb 5, 2018

@heuermh

This comment has been minimized.

Member

heuermh commented Feb 5, 2018

Closing, will separate this into two PRs.

@heuermh heuermh closed this Feb 5, 2018

@AmplabJenkins

This comment has been minimized.

AmplabJenkins commented Feb 5, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/cannoli-prb/95/

Build result: ABORTED

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/cannoli.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/100/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 549e010 # timeout=10Checking out Revision 549e010 (origin/pr/100/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 549e010d5c0b50c76381433021fb324e9013df70First time build. Skipping changelog.Triggering cannoli-prb ? 2.6.0,2.10,2.0.0,centosTriggering cannoli-prb ? 2.3.0,2.11,1.6.1,centosTriggering cannoli-prb ? 2.3.0,2.10,1.6.1,centosTriggering cannoli-prb ? 2.6.0,2.11,1.6.1,centosTriggering cannoli-prb ? 2.3.0,2.10,2.0.0,centosTriggering cannoli-prb ? 2.6.0,2.10,1.6.1,centosTriggering cannoli-prb ? 2.6.0,2.11,2.0.0,centosTriggering cannoli-prb ? 2.3.0,2.11,2.0.0,centoscannoli-prb ? 2.6.0,2.10,2.0.0,centos completed with result ABORTEDcannoli-prb ? 2.3.0,2.11,1.6.1,centos completed with result ABORTEDcannoli-prb ? 2.3.0,2.10,1.6.1,centos completed with result ABORTEDcannoli-prb ? 2.6.0,2.11,1.6.1,centos completed with result ABORTEDcannoli-prb ? 2.3.0,2.10,2.0.0,centos completed with result ABORTEDcannoli-prb ? 2.6.0,2.10,1.6.1,centos completed with result ABORTEDcannoli-prb ? 2.6.0,2.11,2.0.0,centos completed with result ABORTEDcannoli-prb ? 2.3.0,2.11,2.0.0,centos completed with result ABORTEDNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh heuermh deleted the heuermh:drop-spark1x-scala210 branch Feb 8, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment