Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MemSQL Interface on CDH Spark over Yarn #13

Closed
0x0ece opened this issue Mar 7, 2016 · 3 comments
Closed

MemSQL Interface on CDH Spark over Yarn #13

0x0ece opened this issue Mar 7, 2016 · 3 comments

Comments

@0x0ece
Copy link
Contributor

0x0ece commented Mar 7, 2016

Hi,

I'm trying to run memsql-spark-interface-1.3.0 on a CDH 5.1 Spark cluster, in yarn (client) mode.

For context, I can run spark-shell, and I can include memsql interface jar and play with it. That works fine.

spark-shell --master yarn --jars memsql-spark-interface-1.3.0.jar

If I try to run the application, though, I get this error:

spark-submit --master yarn memsql-spark-interface-1.3.0.jar --port 10009 --dataDir /tmp/memsql/ --dbHost $MASTER_IP

16/03/07 21:49:07 INFO Submitted application application_1456615046564_0024
16/03/07 21:49:15 INFO Connected to a Spark master on yarn-client
Uncaught error from thread [spark-interface-akka.actor.default-dispatcher-4] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[spark-interface]
java.lang.AbstractMethodError
    at akka.actor.ActorLogging$class.$init$(Actor.scala:335)
    at spray.can.HttpManager.<init>(HttpManager.scala:29)
    at spray.can.HttpExt$$anonfun$1.apply(Http.scala:153)
    at spray.can.HttpExt$$anonfun$1.apply(Http.scala:153)
    at akka.actor.TypedCreatorFunctionConsumer.produce(Props.scala:401)
    at akka.actor.Props.newActor(Props.scala:339)
    at akka.actor.ActorCell.newActor(ActorCell.scala:534)
    at akka.actor.ActorCell.create(ActorCell.scala:560)
    at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:425)
    at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447)
    at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)
    at akka.dispatch.Mailbox.run(Mailbox.scala:218)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [03/07/2016 21:49:15.509] [spark-interface-akka.actor.default-dispatcher-4] [ActorSystem(spark-interface)] Uncaught error from thread [spark-interface-akka.actor.default-dispatcher-4] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.AbstractMethodError
    at akka.actor.ActorLogging$class.$init$(Actor.scala:335)
    at spray.can.HttpManager.<init>(HttpManager.scala:29)
    at spray.can.HttpExt$$anonfun$1.apply(Http.scala:153)
    at spray.can.HttpExt$$anonfun$1.apply(Http.scala:153)
    at akka.actor.TypedCreatorFunctionConsumer.produce(Props.scala:401)
    at akka.actor.Props.newActor(Props.scala:339)
    at akka.actor.ActorCell.newActor(ActorCell.scala:534)
    at akka.actor.ActorCell.create(ActorCell.scala:560)
    at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:425)
    at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447)
    at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)
    at akka.dispatch.Mailbox.run(Mailbox.scala:218)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[ERROR] [03/07/2016 21:49:15.557] [spark-interface-akka.actor.default-dispatcher-3] [akka://spark-interface/user/IO-HTTP] head of empty list
java.util.NoSuchElementException: head of empty list
    at scala.collection.immutable.Nil$.head(List.scala:337)
    at scala.collection.immutable.Nil$.head(List.scala:334)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
    at akka.actor.ActorCell.invoke(ActorCell.scala:456)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
    at akka.dispatch.Mailbox.run(Mailbox.scala:219)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[ERROR] [03/07/2016 21:49:15.561] [spark-interface-akka.actor.default-dispatcher-3] [akka://spark-interface/user/IO-HTTP] changing Recreate into Create after java.util.NoSuchElementException: head of empty list
Uncaught error from thread [spark-interface-akka.actor.default-dispatcher-3] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[spark-interface]
java.lang.AbstractMethodError
    at akka.actor.ActorLogging$class.$init$(Actor.scala:335)
    at spray.can.HttpManager.<init>(HttpManager.scala:29)
    at spray.can.HttpExt$$anonfun$1.apply(Http.scala:153)
    at spray.can.HttpExt$$anonfun$1.apply(Http.scala:153)
    at akka.actor.TypedCreatorFunctionConsumer.produce(Props.scala:401)
    at akka.actor.Props.newActor(Props.scala:339)
    at akka.actor.ActorCell.newActor(ActorCell.scala:534)
    at akka.actor.ActorCell.create(ActorCell.scala:560)
    at akka.actor.dungeon.FaultHandling$class.finishCreate(FaultHandling.scala:135)
    at akka.actor.dungeon.FaultHandling$class.faultCreate(FaultHandling.scala:129)
    at akka.actor.ActorCell.faultCreate(ActorCell.scala:338)
    at akka.actor.dungeon.FaultHandling$class.faultRecreate(FaultHandling.scala:58)
[ERROR] [03/07/2016 21:49:15.564] [spark-interface-akka.actor.default-dispatcher-3] [ActorSystem(spark-interface)] Uncaught error from thread [spark-interface-akka.actor.default-dispatcher-3] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.AbstractMethodError
    at akka.actor.ActorLogging$class.$init$(Actor.scala:335)
    at spray.can.HttpManager.<init>(HttpManager.scala:29)
    at spray.can.HttpExt$$anonfun$1.apply(Http.scala:153)
    at spray.can.HttpExt$$anonfun$1.apply(Http.scala:153)
    at akka.actor.TypedCreatorFunctionConsumer.produce(Props.scala:401)
    at akka.actor.Props.newActor(Props.scala:339)
    at akka.actor.ActorCell.newActor(ActorCell.scala:534)
    at akka.actor.ActorCell.create(ActorCell.scala:560)
    at akka.actor.dungeon.FaultHandling$class.finishCreate(FaultHandling.scala:135)
    at akka.actor.dungeon.FaultHandling$class.faultCreate(FaultHandling.scala:129)
    at akka.actor.ActorCell.faultCreate(ActorCell.scala:338)
    at akka.actor.dungeon.FaultHandling$class.faultRecreate(FaultHandling.scala:58)
    at akka.actor.ActorCell.faultRecreate(ActorCell.scala:338)
    at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:428)
    at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447)
    at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)
    at akka.dispatch.Mailbox.run(Mailbox.scala:218)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
    at akka.actor.ActorCell.faultRecreate(ActorCell.scala:338)

    at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:428)
    at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447)
    at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)
    at akka.dispatch.Mailbox.run(Mailbox.scala:218)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I have the feeling this is due to CDH, but I wanted to double check if memsql interface can work in yarn mode.

Googling around, I've found this similar issue, in a totally unrelated project:
FRosner/spawncamping-dds#47

Thanks, E.
@choochootrain
Copy link

hey emanuele! to the best of my knowledge, a generic spark app shouldn't need to do anything special to run on yarn. it looks like the issue is that CDH and spray using different versions of akka.

there's a relevant issue in spark-jobserver: spark-jobserver/spark-jobserver#60 with a potential workaround at the bottom of the ticket. hope that helps 😄

@0x0ece
Copy link
Contributor Author

0x0ece commented Mar 7, 2016

Thanks, I'll see if I can get it working...

@0x0ece
Copy link
Contributor Author

0x0ece commented Mar 8, 2016

I think I got it running. Pretty involuted, but definitely an issue with CDH.

For future reference, this is the config to run it with Spark on CDH 5.1.

-lazy val akkaVersion = "2.3.9"
-lazy val sprayVersion = "1.3.2"
+lazy val akkaVersion = "2.2.3"
+lazy val sprayVersion = "1.2.3"
+lazy val sprayJsonVersion = "1.3.2"
...
-        "io.spray" %% "spray-json" % sprayVersion,
-        "io.spray" %% "spray-can" % sprayVersion,
-        "io.spray" %% "spray-routing" % sprayVersion,
+        "io.spray" %% "spray-json" % sprayJsonVersion,
+        "io.spray" % "spray-can" % sprayVersion,
+        "io.spray" % "spray-routing" % sprayVersion,

(note the single vs double % due to older versions of spray -- also note that spray-json versioning is unrelated to spray-*)

Time permitting I'll see if I can have it working with a jarjar approach too.

@0x0ece 0x0ece closed this as completed Mar 8, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@choochootrain @0x0ece