Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClassNotFoundException: EsHadoopNoNodesLeftException #585

Closed
freakinruben opened this issue Oct 30, 2015 · 6 comments

Comments

@freakinruben
Copy link

commented Oct 30, 2015

I've created a spark job that reads from elasticsearch and then writes back to it. Locally it runs fine, but when I submit it to a cluster I'm not able to get it working. I've tried different things, as described below.

Spark-cluster is running spark-1.5.1-bin-without-hadoop on 3 nodes as a test setup.

I included the following packages in my uberjar:

  • org.elasticsearch/elasticsearch-spark_2.10: 2.2.0-m1 (also tried with 2.1.1)
  • org.elasticsearch/elasticsearch-hadoop-mr: 2.2.0-m1

The job then fails with

java.lang.ClassNotFoundException: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException

The ClassNotFoundException surprised me, it also prevents me from seeing the
actual reason for the exception, so I tried to solve the ClassNotFoundException.

I downloaded the packages on all nodes, added them to the spark classpath, rebooted
spark and re-submitted the job.

It then tells me

15/10/30 10:57:42 WARN scheduler.TaskSetManager:
Lost task 0.0 in stage 5.0 (TID 11, 10.20.0.97):
java.lang.Error: Multiple ES-Hadoop versions detected in the classpath; please use only one
....

Makes sense, there are multiple ES-Hadoop packages.. So I build an uberjar without
the packages described in step 1.

Exception in thread "main" java.lang.ExceptionInInitializerError
....
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.spark.rdd.api.java.JavaEsSpark

This also happens when I don't put the packages on the spark-classpath but provide
them when submitting the job:

  • submitted with --packages org.elasticsearch:elasticsearch-hadoop-mr:2.2.0-m1,org.elasticsearch:elasticsearch-spark_2.10:2.2.0-m1
  • submitted with --jars /home/hadoop/extra/elasticsearch-spark_2.10-2.2.0-m1.jar,/home/hadoop/extra/elasticsearch-hadoop-mr-2.2.0-m1.jar

So now I'm a bit confused of what's the right way to go about this. Do you have
any tips on how to run the job?

@costin

This comment has been minimized.

Copy link
Member

commented Oct 30, 2015

Why do you submit both -hadoop-mr and elasticsearch-spark jars? You only need the spark jar - that's it; it contains both mr and the spark native API.

P.S. The ClassNotFoundException appears because Spark will shutdown its job classloader immediately in case of an exception so any other classes that need to be loaded, will fail causing the initial error to be hidden.

@costin

This comment has been minimized.

Copy link
Member

commented Oct 30, 2015

And by the way, 2.2.0-beta1 was released yesterday; try that instead.

@costin costin added the question label Oct 30, 2015

@freakinruben

This comment has been minimized.

Copy link
Author

commented Nov 6, 2015

Hey Costin,

Thanks for the answers, the ClassNotFoundException was the confusing part, at which I decided to add the hadoop-mr in the hope the class would become available.
I found the actual error in one of the workers and it works good now, also with 2.2.0-beta1. Thanks for your help!

@fabiantan

This comment has been minimized.

Copy link

commented Nov 12, 2015

freakinruben@ looks like im having the same issue, however, im still troubleshooting it. Can you share what was the actual error in the worker?

@costin

This comment has been minimized.

Copy link
Member

commented Nov 12, 2015

@fabiantan

java.lang.Error: Multiple ES-Hadoop versions detected in the classpath; please use only one

@freakinruben

This comment has been minimized.

Copy link
Author

commented Nov 13, 2015

@fabiantan The original error (where all of this started) had to do with writing to ES with a RDD that didn't have the configuration of where to find ES

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.