Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/package$ #295

Closed
lairdnote opened this issue Oct 11, 2014 · 6 comments
Closed

Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/package$ #295

lairdnote opened this issue Oct 11, 2014 · 6 comments
Labels
invalid :Spark v2.1.0.Beta3

Comments

@lairdnote
Copy link

Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/package$
at spark.c4hcdn.KafkaNetWork$.main(KafkaNetWork.scala:52)
at spark.c4hcdn.KafkaNetWork.main(KafkaNetWork.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.spark.package$
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 9 more

val esRDD = lines.flatMap(_.split("\\n"))
              .map(x => p.parseRecord(x))
              .map(x => Map("clientIpAddress" -> Utils.clientIpAddress(x), "dateTime" -> Utils.dateTime(x), "request" -> Utils.request(x), "httpStatusCode" -> Utils.httpStatusCode(x), "bytesSent"-> Utils.bytesSent(x), "referer"-> Utils.referer(x), "userAgent" -> Utils.userAgent(x), "hit"-> Utils.hit(x), "origintime"-> Utils.origintime(x)))

 sc.makeRDD(Seq(esRDD)).saveToEs("spark/docs")

import org.elasticsearch.spark._

where are error?
@lairdnote lairdnote changed the title Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/package$ at spark.c4hcdn.KafkaNetWork$.main(KafkaNetWork.scala:52) at spark.c4hcdn.KafkaNetWork.main(KafkaNetWork.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.elasticsearch.spark.package$ at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/package$ Oct 11, 2014
@costin
Copy link
Member

costin commented Oct 11, 2014

@lairdnote Hi, based on the stacktrace, it looks like you don't have es-hadoop in the runtime classpath. In other words, es-hadoop has not been deployed through-out the nodes where your spark code is running (NoClassDefFound means the class definition has not been found).
You should be able to fix this by making sure es-hadoop.jar is available in the classpath.
Make sure you are using the es-hadoop 2.1.Beta3 jar.

If you need help, I recommend reaching out to the mailing list.

@costin costin closed this as completed Oct 11, 2014
@costin costin added the :Spark label Oct 11, 2014
@lairdnote
Copy link
Author

Thanks you .

@liusztc09
Copy link

Hi @costin,

I got the same problem using elasticsearch.spark. It seems that the jar elasticsearch-spark_2.10_Beta4 includes both es hadoop and es spark. Also, the cause of my problem is due to the call of rdd.saveToEs()

Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/package$
at com.hearst.sample.LoadJsonToEs$.main(LoadJsonToEs.scala:28)
at com.hearst.sample.LoadJsonToEs.main(LoadJsonToEs.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.spark.package$
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 11 more

Could you please have look at the problem?

Thanks,

@costin
Copy link
Member

costin commented Jun 14, 2015

@liusztc09 es-hadoop is not part of the classpath. This is not a bug but a configuration issue on your side. This has been explained in the bug above (closed and marked as invalid) as well as the fact that this is an issue tracker and for questions and such you should try the mailing list the forum.

@liusztc09
Copy link

@costin Thanks a lot for the explanation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
invalid :Spark v2.1.0.Beta3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@costin @lairdnote @liusztc09 and others