Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
ClassNotFoundException: EsHadoopNoNodesLeftException #585
I've created a spark job that reads from elasticsearch and then writes back to it. Locally it runs fine, but when I submit it to a cluster I'm not able to get it working. I've tried different things, as described below.
Spark-cluster is running spark-1.5.1-bin-without-hadoop on 3 nodes as a test setup.
I included the following packages in my uberjar:
The job then fails with
I downloaded the packages on all nodes, added them to the spark classpath, rebooted
It then tells me
Makes sense, there are multiple ES-Hadoop packages.. So I build an uberjar without
This also happens when I don't put the packages on the spark-classpath but provide
So now I'm a bit confused of what's the right way to go about this. Do you have
Why do you submit both -hadoop-mr and elasticsearch-spark jars? You only need the spark jar - that's it; it contains both mr and the spark native API.
Thanks for the answers, the