Caused by: java.io.IOException: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[10.XXX.XXX.XX:9200]]

[senthilkaconnect]

I am using HDP-2.3.0 with ElasticSearch-2.1 in AWS Cloud.

Driver: elasticsearch-hadoop-2.2.0.BUILD-SNAPSHOT.jar

I have created a Hive Table with below schema:
CREATE EXTERNAL TABLE sample_data_es_target_final_20
(accept_language String,
browser string,
xxxx String,
......)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'sample-testing/testing', 'es.nodes' = '10.XXX.XXX.XX', 'es.nodes.wan.only'='true');

our ES Cluster consists of more than 6 nodes (master + Data) and one client Node as of now.
I have ingested more than data into this table but not able to query from it.

Example of queries for reading from it

1. select count(*) from sample_data_es_target_final_20;
2. select * from sample_data_es_target_final_20 limit 1;

Issue:
Caused by: java.io.IOException: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[10.xxx.xxx.xx:9200]]
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:141)
at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:113)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:61)
... 16 more
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[10.177.228.19:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:383)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:375)
at org.elasticsearch.hadoop.rest.RestClient.scan(RestClient.java:432)
at org.elasticsearch.hadoop.rest.RestRepository.scanLimit(RestRepository.java:139)
at org.elasticsearch.hadoop.rest.QueryBuilder.build(QueryBuilder.java:181)
at org.elasticsearch.hadoop.mr.EsInputFormat$ShardRecordReader.next(EsInputFormat.java:290)
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more

[costin]

@senthilkaconnect Please post your question on the forum not on the issue tracker. Also see this page regarding on the needed information to help diagnose.

The error above is quite common and stems typically from network misconfiguration.