New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException #963

Closed
zlkkuu opened this Issue Apr 25, 2017 · 1 comment

Comments

Projects
None yet
2 participants
@zlkkuu

zlkkuu commented Apr 25, 2017

I build es(Version 2.4) cluster in AWS EC2, Spark(2.0) in EMR
It is always running ok, but sometimes the application is failed.
While restart the application of the spark, write data to es is ok.
The maven:

org.elasticsearch
elasticsearch-spark-20_2.10
5.2.2


org.apache.spark
spark-core_2.10
2.0.0

The code is:
val conf = new SparkConf().setAppName("EsClick")
conf.set("es.nodes", "es-local.kiwisns.com")
conf.set("es.nodes.wan.only", "true")
conf.set("es.port", "9200")

val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
sqlContext.createDataFrame(rdd.map(x => new String(x.getPayload, "UTF-8"))
.flatMap(checkClick), affiliate_data_schema).where(s"dt = '${lastDay}'")
.saveToEs(s"spark-aflt-all-record-${lastDayType}/record")

The error is:
17/04/24 06:32:00 INFO TaskSetManager: Starting task 20.0 in stage 11.0 (TID 219, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 20, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Starting task 0.0 in stage 11.0 (TID 220, ip-10-255-194-8.us-west-1.compute.internal, executor 1, partition 0, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Starting task 21.0 in stage 11.0 (TID 221, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 21, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Starting task 1.0 in stage 11.0 (TID 222, ip-10-255-194-8.us-west-1.compute.internal, executor 1, partition 1, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on ip-10-255-194-8.us-west-1.compute.internal:42269 (size: 13.2 KB, free: 1615.9 MB)
17/04/24 06:32:00 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on ip-10-129-11-143.us-west-1.compute.internal:43272 (size: 13.2 KB, free: 1615.9 MB)
17/04/24 06:32:00 INFO TaskSetManager: Starting task 22.0 in stage 11.0 (TID 223, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 22, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Starting task 23.0 in stage 11.0 (TID 224, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 23, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Starting task 24.0 in stage 11.0 (TID 225, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 24, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 WARN TaskSetManager: Lost task 20.0 in stage 11.0 (TID 219, ip-10-129-11-143.us-west-1.compute.internal, executor 2): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:199)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:542)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:40)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[es-local.kiwisns.com:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:431)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:411)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:415)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:141)
at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:596)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:192)
... 10 more

17/04/24 06:32:00 INFO TaskSetManager: Lost task 21.0 in stage 11.0 (TID 221) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 1]
17/04/24 06:32:00 INFO TaskSetManager: Lost task 23.0 in stage 11.0 (TID 224) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 2]
17/04/24 06:32:00 INFO TaskSetManager: Lost task 22.0 in stage 11.0 (TID 223) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 3]
17/04/24 06:32:00 INFO TaskSetManager: Starting task 22.1 in stage 11.0 (TID 226, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 22, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Starting task 23.1 in stage 11.0 (TID 227, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 23, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Lost task 24.0 in stage 11.0 (TID 225) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 4]
17/04/24 06:32:00 INFO TaskSetManager: Starting task 24.1 in stage 11.0 (TID 228, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 24, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Lost task 22.1 in stage 11.0 (TID 226) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 5]
17/04/24 06:32:00 INFO TaskSetManager: Starting task 22.2 in stage 11.0 (TID 229, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 22, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Lost task 23.1 in stage 11.0 (TID 227) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 6]
17/04/24 06:32:00 INFO TaskSetManager: Starting task 23.2 in stage 11.0 (TID 230, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 23, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Lost task 24.1 in stage 11.0 (TID 228) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 7]
17/04/24 06:32:00 INFO TaskSetManager: Starting task 24.2 in stage 11.0 (TID 231, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 24, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Lost task 22.2 in stage 11.0 (TID 229) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 8]
17/04/24 06:32:00 INFO TaskSetManager: Starting task 22.3 in stage 11.0 (TID 232, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 22, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Lost task 23.2 in stage 11.0 (TID 230) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 9]
17/04/24 06:32:00 INFO TaskSetManager: Starting task 23.3 in stage 11.0 (TID 233, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 23, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Lost task 24.2 in stage 11.0 (TID 231) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 10]
17/04/24 06:32:00 INFO TaskSetManager: Starting task 24.3 in stage 11.0 (TID 234, ip-10-129-11-143.us-west-1.compute.internal, executor 2, partition 24, NODE_LOCAL, 6314 bytes)
17/04/24 06:32:00 INFO TaskSetManager: Lost task 22.3 in stage 11.0 (TID 232) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 11]
17/04/24 06:32:00 ERROR TaskSetManager: Task 22 in stage 11.0 failed 4 times; aborting job
17/04/24 06:32:00 INFO TaskSetManager: Lost task 23.3 in stage 11.0 (TID 233) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 12]
17/04/24 06:32:00 INFO YarnClusterScheduler: Cancelling stage 11
17/04/24 06:32:00 INFO YarnClusterScheduler: Stage 11 was cancelled
17/04/24 06:32:00 INFO DAGScheduler: ResultStage 11 (runJob at EsSparkSQL.scala:76) failed in 0.166 s due to Job aborted due to stage failure: Task 22 in stage 11.0 failed 4 times, most recent failure: Lost task 22.3 in stage 11.0 (TID 232, ip-10-129-11-143.us-west-1.compute.internal, executor 2): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:199)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:542)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:40)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[es-local.kiwisns.com:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:431)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:411)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:415)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:141)
at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:596)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:192)
... 10 more

Driver stacktrace:
17/04/24 06:32:00 INFO TaskSetManager: Lost task 24.3 in stage 11.0 (TID 234) on ip-10-129-11-143.us-west-1.compute.internal, executor 2: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException (Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only') [duplicate 13]
17/04/24 06:32:00 INFO DAGScheduler: Job 10 failed: runJob at EsSparkSQL.scala:76, took 0.181102 s
17/04/24 06:32:00 INFO JobScheduler: Finished job streaming job 1493040720000 ms.0 from job set of time 1493040720000 ms
17/04/24 06:32:00 INFO JobScheduler: Total delay: 0.280 s for time 1493040720000 ms (execution: 0.274 s)
17/04/24 06:32:00 INFO UnionRDD: Removing RDD 24 from persistence list
17/04/24 06:32:00 INFO BlockManager: Removing RDD 24
17/04/24 06:32:00 INFO BlockRDD: Removing RDD 22 from persistence list
17/04/24 06:32:00 INFO BlockManager: Removing RDD 22
17/04/24 06:32:00 INFO PluggableInputDStream: Removing blocks of RDD BlockRDD[22] at receiverStream at ReceiverLauncher.java:114 of time 1493040720000 ms
17/04/24 06:32:00 ERROR JobScheduler: Error running job streaming job 1493040720000 ms.0
org.apache.spark.SparkException: Job aborted due to stage failure: Task 22 in stage 11.0 failed 4 times, most recent failure: Lost task 22.3 in stage 11.0 (TID 232, ip-10-129-11-143.us-west-1.compute.internal, executor 2): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:199)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:542)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:40)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[es-local.kiwisns.com:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:431)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:411)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:415)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:141)
at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:596)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:192)
... 10 more

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1931)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)
at org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:76)
at org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:58)
at org.elasticsearch.spark.sql.package$SparkDataFrameFunctions.saveToEs(package.scala:30)
at papaya.esClick$$anonfun$main$2.apply(esClick.scala:158)
at papaya.esClick$$anonfun$main$2.apply(esClick.scala:149)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:415)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:254)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:254)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:254)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:199)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:542)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:40)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
... 3 more
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[es-local.kiwisns.com:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:431)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:411)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:415)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:141)
at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:596)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:192)
... 10 more
17/04/24 06:32:00 INFO BlockManagerInfo: Removed input-0-1493040553606 on ip-10-255-194-8.us-west-1.compute.internal:42269 in memory (size: 252.8 KB, free: 1616.2 MB)
17/04/24 06:32:00 INFO BlockManagerInfo: Removed input-0-1493040553607 on ip-10-255-194-8.us-west-1.compute.internal:42269 in memory (size: 86.6 KB, free: 1616.2 MB)
17/04/24 06:32:00 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 22 in stage 11.0 failed 4 times, most recent failure: Lost task 22.3 in stage 11.0 (TID 232, ip-10-129-11-143.us-west-1.compute.internal, executor 2): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:199)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:542)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:40)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[es-local.kiwisns.com:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:431)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:411)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:415)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:141)
at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:596)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:192)
... 10 more

Driver stacktrace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 22 in stage 11.0 failed 4 times, most recent failure: Lost task 22.3 in stage 11.0 (TID 232, ip-10-129-11-143.us-west-1.compute.internal, executor 2): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:199)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:542)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:40)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[es-local.kiwisns.com:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:431)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:411)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:415)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:141)
at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:596)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:192)
... 10 more

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1931)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)
at org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:76)
at org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:58)
at org.elasticsearch.spark.sql.package$SparkDataFrameFunctions.saveToEs(package.scala:30)
at papaya.esClick$$anonfun$main$2.apply(esClick.scala:158)
at papaya.esClick$$anonfun$main$2.apply(esClick.scala:149)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:415)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:254)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:254)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:254)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:199)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:542)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:40)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
... 3 more
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[es-local.kiwisns.com:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:431)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:411)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:415)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:141)
at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:596)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:192)
... 10 more
17/04/24 06:32:00 INFO ZkState: Starting curator service
17/04/24 06:32:00 INFO CuratorFrameworkImpl: Starting
17/04/24 06:32:00 INFO ZooKeeper: Initiating client connection, connectString=172.16.1.162:2181,172.16.1.163:2181,172.16.1.164:2181 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@6cb4e30b
17/04/24 06:32:00 INFO BlockRDD: Removing RDD 23 from persistence list
17/04/24 06:32:00 INFO ClientCnxn: Opening socket connection to server 172.16.1.163/172.16.1.163:2181. Will not attempt to authenticate using SASL (unknown error)
17/04/24 06:32:00 INFO BlockManagerInfo: Removed input-0-1493040553608 on ip-10-255-194-8.us-west-1.compute.internal:42269 in memory (size: 191.6 KB, free: 1616.4 MB)
17/04/24 06:32:00 INFO BlockManagerInfo: Removed input-0-1493040553609 on ip-10-255-194-8.us-west-1.compute.internal:42269 in memory (size: 313.0 KB, free: 1616.7 MB)
17/04/24 06:32:00 INFO ClientCnxn: Socket connection established to 172.16.1.163/172.16.1.163:2181, initiating session
17/04/24 06:32:00 INFO PluggableInputDStream: Removing blocks of RDD BlockRDD[23] at receiverStream at ReceiverLauncher.java:114 of time 1493040720000 ms
17/04/24 06:32:00 INFO BlockManagerInfo: Removed input-0-1493040553610 on ip-10-255-194-8.us-west-1.compute.internal:42269 in memory (size: 107.4 KB, free: 1616.8 MB)
17/04/24 06:32:00 INFO BlockManagerInfo: Removed input-0-1493040553611 on ip-10-255-194-8.us-west-1.compute.internal:42269 in memory (size: 259.1 KB, free: 1617.1 MB)
17/04/24 06:32:00 INFO ClientCnxn: Session establishment complete on server 172.16.1.163/172.16.1.163:2181, sessionid = 0x2535a6cba5f5216, negotiated timeout = 120000
17/04/24 06:32:00 INFO BlockManager: Removing RDD 23
17/04/24 06:32:00 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 22 in stage 11.0 failed 4 times, most recent failure: Lost task 22.3 in stage 11.0 (TID 232, ip-10-129-11-143.us-west-1.compute.internal, executor 2): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:199)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:542)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:40)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:76)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[es-local.kiwisns.com:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:431)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:411)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:415)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:141)
at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:596)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:192)
... 10 more

@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Apr 25, 2017

Contributor

@zlkkuu We prefer it if you would post questions like this on our discuss forums since we reserve Github for bug reports and feature tracking only. Thank you!

Contributor

jbaiera commented Apr 25, 2017

@zlkkuu We prefer it if you would post questions like this on our discuss forums since we reserve Github for bug reports and feature tracking only. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment