Unable to start the job server on CDH 5.5.2 cluster containing spark 1.5.0 #394
Comments
|
Ok, Spark 1.5.0 Would you be able to list out the dependencies or library jars included with CDH? CDH often replaces the version of various libraries, such as Akka, which is probably why you see the error below.
|
|
velvia - CDH 5.5.2 uses akka 2.3.4. Please see here - http://archive.cloudera.com/cdh5/cdh/5/spark-1.5.0-cdh5.5.2.releasenotes.html |
|
Yup that's right the release notes say that CDH 5.5.2 comes with akka 2.3.4 but, I guess they have reverted the akka upgrade that was supposed to come with CDH 5.5.2 and CDH still comes with 2.2.3 version of akka. |
|
Hmmm. I think we’ll need to have a branch of the current job server with Akka downgraded then. The other route is to try to shade the Akka jar. It didn’t work before, but maybe it will work now.
|
|
@rohankalra91 are you still having issues? |
|
Hi, I am seeing the same issue on Cloudera 5.7 any pointers on how I can fix this? Getting the same error: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/spark) overrides detected (/home/appml/sparkhome). |
|
Cloudera is using akka version 2.2.3 and Versions.scala had 2.2.6. It appears like akka version mismatch was causing this issue so I downgraded akka version in Versions.scala and compiled it again. Getting the following compilation error on cluster.subscribe: [success] created output: /home/appml/spark-jobserver-master/job-server/target |
|
Yeah unfortunately it's not easy to downgrade to Akka 2.2.x anymore. The only real solution is to shade Akka, but this is not easy due to all of On Mon, Jul 18, 2016 at 3:13 PM, aniruddh02 notifications@github.com
If you are free, you need to free somebody else. |
|
Hi, |
|
@koettert Thanks! I will take a look at this and will let you know if it worked. Thanks, |
|
Thanks Tobias. I wonder if there's any way to make this generic or easier to build for others. -Evan
|
|
I tried https://github.com/bjoernlohrmann/spark-jobserver as suggested by Tobias and faced same issue. |
|
Hi @koettert The link eemss broken. Do you know if he provides a working versión for CDH 5.7? |
|
Hi @velvia @aniruddh02 @koettert update: the repo of @koettert works with CDH 5.7 and jobserver 0.6.2. Thanks alot. |
|
I am getting same error with CDH-5.8.2 Spark and jobserver 0.6.2. |
|
Closing due to inactivity. Please reopen if you still have problems. |
I have successfully deployed the job server on my cluster using "bin/server_deploy.sh qa" command , but when I try to start the server using command "./server_start.sh" on my host machine, then it gives the following exception:-
Exception in thread "main" java.lang.NoSuchMethodError: akka.util.Helpers$.ConfigOps(Lcom/typesafe/config/Config;)Lcom/typesafe/config/Config;
at akka.cluster.ClusterSettings.(ClusterSettings.scala:27)
at akka.cluster.Cluster.(Cluster.scala:67)
at akka.cluster.Cluster$.createExtension(Cluster.scala:42)
at akka.cluster.Cluster$.createExtension(Cluster.scala:37)
at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:654)
at akka.actor.ExtensionId$class.apply(Extension.scala:79)
at akka.cluster.Cluster$.apply(Cluster.scala:37)
at akka.cluster.ClusterActorRefProvider.createRemoteWatcher(ClusterActorRefProvider.scala:66)
at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:186)
at akka.cluster.ClusterActorRefProvider.init(ClusterActorRefProvider.scala:58)
at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
at spark.jobserver.JobServer$.spark$jobserver$JobServer$$makeSupervisorSystem$1(JobServer.scala:128)
at spark.jobserver.JobServer$$anonfun$main$1.apply(JobServer.scala:130)
at spark.jobserver.JobServer$$anonfun$main$1.apply(JobServer.scala:130)
at spark.jobserver.JobServer$.start(JobServer.scala:54)
at spark.jobserver.JobServer$.main(JobServer.scala:130)
at spark.jobserver.JobServer.main(JobServer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
This is how my qa.sh looks like:-
DEPLOY_HOSTS=""
APP_USER=cloudera-scm
APP_GROUP=cloudera-scm
// SSH Key to login to deploy server
SSH_KEY=/home/ubuntu/Downloads/Cloudera.pem
INSTALL_DIR=/home/cloudera-scm/spark-jobserver-deploy
LOG_DIR=/var/log/job-server
PIDFILE=spark-jobserver.pid
JOBSERVER_MEMORY=1G
SPARK_VERSION=1.5.0
SPARK_HOME=/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/spark
SPARK_CONF_DIR=$SPARK_HOME/conf
// Only needed for Mesos deploys
//SPARK_EXECUTOR_URI=/home/spark/spark-1.6.0.tar.gz
// Only needed for YARN running outside of the cluster
// You will need to COPY these files from your cluster to the remote machine
// Normally these are kept on the cluster in /etc/hadoop/conf
YARN_CONF_DIR=/etc/hadoop/conf
HADOOP_CONF_DIR=/etc/hadoop/conf
// Also optional: extra JVM args for spark-submit
// export SPARK_SUBMIT_OPTS+="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5433"
SCALA_VERSION=2.10.4 # or 2.11.6
And this is how my qa.conf looks like:-
// Template for a Spark Job Server configuration file
// When deployed these settings are loaded when job server starts
// Spark Cluster / Job Server configuration
spark {
// spark.master will be passed to each job's JobContext
// master = "local[4]"
// master = "mesos://vm28-hulk-pub:5050"
master = "yarn-client"
// Default no. of CPUs for jobs to use for Spark standalone cluster
job-number-cpus = 4
jobserver {
port = 8090
jar-store-rootdir = /tmp/jobserver/jars
}
// predefined Spark contexts
// contexts {
// my-low-latency-context {
// num-cpu-cores = 1 # Number of cores to allocate. Required.
// memory-per-node = 512m # Executor memory per node, -Xmx style eg 512m, 1G, etc.
// }
// define additional contexts here
// }
// universal context configuration. These settings can be overridden, see README.md
context-settings {
num-cpu-cores = 2 # Number of cores to allocate. Required.
memory-per-node = 512m # Executor memory per node, -Xmx style eg 512m, #1G, etc.
// such as hadoop connection settings that don't use the "spark." prefix
passthrough {
//es.nodes = "192.1.1.1"
}
}
// This needs to match SPARK_HOME for cluster SparkContexts to be created successfully
// home = "/home/spark/spark"
}
// Note that you can use this file to define settings not only for job server,
// but for your Spark jobs as well. Spark job configuration merges with this configuration file as defaults.
akka {
remote.netty.tcp {
// This controls the maximum message size, including job results, that can be sent
// maximum-frame-size = 10 MiB
}
}
spray.can {
server {
parsing {
max-content-length = 100m
}
}
}
However, I had successfully deployed and started the server on my local machine which contains spark 1.5.1 using "bin/server_deploy.sh development" command.
The text was updated successfully, but these errors were encountered: