Apache Spark application fails when mysql-async is add in build.sbt

[rsiebeling]

My application developed using Apache Spark is failing when the mysql-async library is add to the build.sbt.

I've add the following line to build.sbt "com.github.mauricio" %% "mysql-async" % "0.2.11"
When this line is commented out the application runs just fine.

Could you please help?

I'm getting the following error message:

23:43:54.429 [spark-akka.actor.default-dispatcher-3] INFO o.a.s.s.local.LocalTaskSetManager - Loss was due to java.lang.ClassNotFoundException
java.lang.ClassNotFoundException: scala.None$
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:36)
at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
at java.io.ObjectInputStream.readClassDesc(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39)
at org.apache.spark.scheduler.ResultTask$.deserializeInfo(ResultTask.scala:61)
at org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:129)
at java.io.ObjectInputStream.readExternalData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:61)
at org.apache.spark.scheduler.local.LocalScheduler.runTask(LocalScheduler.scala:191)
at org.apache.spark.scheduler.local.LocalActor$$anonfun$launchTask$1$$anon$1.run(LocalScheduler.scala:68)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
23:43:54.438 [DAGScheduler] DEBUG o.a.spark.scheduler.DAGScheduler - Got event of type org.apache.spark.scheduler.TaskSetFailed
23:43:54.443 [test-akka.actor.default-dispatcher-3] INFO o.a.spark.scheduler.DAGScheduler - Failed to run count at DataSession.scala:26
23:43:54.447 [spark-akka.actor.default-dispatcher-3] INFO o.a.s.scheduler.local.LocalScheduler - Remove TaskSet 0.0 from pool
[ERROR] [01/19/2014 23:43:54.455] [test-akka.actor.default-dispatcher-6] [akka://test/user/testServer/1/771192171] Job failed: Task 0.0:0 failed more than 4 times; aborting job java.lang.ClassNotFoundException: scala.None$
org.apache.spark.SparkException: Job failed: Task 0.0:0 failed more than 4 times; aborting job java.lang.ClassNotFoundException: scala.None$
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:761)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:759)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:759)
at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:380)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:442)
at org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:150)

[mauricio]

Is this a Scala application?

The error says java.lang.ClassNotFoundException: scala.None$, you're missing Scala's standard library in your classpath.

[rsiebeling]

Yes, it is a Scala application, it was working fine when I did not include the mysql-async library, all actors were up and were communicating with each other.

I would really like to use your mysql-async library if possible... Could it be that there are duplicated dependencies or something like that? I'm not that experienced with scala and sbt but I really would like to get this running.

[mauricio]

Yup, you could have two different versions of the scala library in your classpath. You should check the dependencies SBT is loading and what you have at your classpath.

[rsiebeling]

Ok, thanks for the tip, I will check that

[rsiebeling]

Hi Mauricio,

I solved it, your library required scala 2.10.3 and I was using 2.10.2,
thanks for your help,

kind regards,
Richard

On Mon, Jan 20, 2014 at 1:34 PM, Maurício Linhares <notifications@github.com

wrote:

Closed #81 #81.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/81
.

[mauricio]

Great! 👍