New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doesn't work with ES 5.1 #18

Open
Ksennios opened this Issue Feb 3, 2017 · 0 comments

Comments

Projects
None yet
1 participant
@Ksennios

Ksennios commented Feb 3, 2017

I execute "spark-1.6.2-bin-hadoop2.6\bin\spark-submit --packages TargetHolding/pyspark-elastic:0.4.2
,org.elasticsearch:elasticsearch-spark_2.10:2.4.3 import.py" and have next error:

DAGScheduler: Job 1 failed: runJob at EsSpark.scala:84, took 0.493005 s
Traceback (most recent call last):
File "import.py", line 86, in
main()
File "import.py", line 81, in main
loader.run()
File "import.py", line 57, in run
self.__save_criminal_profiles_to_es()
File "import.py", line 49, in __save_to_es
.saveJsonToEs(self.__args.index)
File ".ivy2\jars\TargetHolding_pyspark-elastic-0.4.2.jar\pyspark_elastic\rdd.py", line 65, in saveJsonToEs
File "spark-1.6.2-bin-hadoop2.6\python\lib\py4j-0.9-src.zip\py4j\java_gateway.py", line 813, in call
File "spark-1.6.2-bin-hadoop2.6\python\lib\py4j-0.9-src.zip\py4j\protocol.py", line 308, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o59.saveJsonToEs.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 2, localhost
): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or
when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:196)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:379)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:40)
at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:84)
at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:84)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 5.1.1
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:191)

With ES 2.3 works without error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment