hadoop trying to connect to elastic search non data node and failed

[tzhang101]

Hi, I am running yarn/spark streaming job and use your package to write to elastic search.
It works mostly but i found the following 2 issues.

1. your package trying to find the AWS private IP's address of the elastic search nodes to connect.
   I will have to do ip forwarding to map the private ip to public ip in order for the write to work.
2. you also find the non-data node in the elastic search cluster to connect and as a result the connection failed. Following is the detailed error message.
   Can you help fix and/or advise how to work around. We need those non-data nodes for security/routing purpose. Thanks.

Tian

15/01/31 03:11:12 INFO httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
15/01/31 03:11:12 INFO httpclient.HttpMethodDirector: Retrying request
15/01/31 03:12:15 INFO httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
15/01/31 03:12:15 INFO httpclient.HttpMethodDirector: Retrying request
15/01/31 03:13:18 INFO httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
15/01/31 03:13:18 INFO httpclient.HttpMethodDirector: Retrying request
15/01/31 03:14:21 ERROR rest.NetworkClient: Node [Connection timed out] failed (10.218.189.172:9200); selected next node [50.16.116.16:9200]

10.218.189.172 is the non-data node and after this failure I can no longer write to elastic search unless I restart the spark streaming job.

[costin]

1. es-hadoop relies on the node http ip - there are various ways to make this work in AWS: use network.publish_address or use some other type of routing/proxying to connect the Spark environment to the Elastic one (again, es-hadoop relies on connecting to each node directly for performance/architectural reasons).
2. What version of es-hadoop are you using and what do these non-data nodes represent? The connector only relies on ES nodes that have an HTTP endpoint and that hold some type of shard information (for both reading and writing).

[tzhang101]

Costin, first thank you so much for responding timely.
We have total 5 nodes, 1 master node, 2 data nodes and 2 client nodes (data=false).
It seems that elasticsearch-hadoop discovered all 5 nodes, but failed to connect to the 2 client nodes.
The reason we have 2 client nodes are for security reason/routing etc. You can see that we are on version 1.4.2 and here is an example of the client node info where I replaced the real IP with X.Y.Z.W.
Thanks.

Tian

    "aHECtVZOTbWDxDnZpMv0pA": {
        "name": "ip-X-Y-Z-W.ec2.internal",
        "transport_address": "inet[/X.Y.Z.W:9300]",
        "host": "ip-X-Y-Z-W.ec2.internal",
        "ip": "X.Y.Z.W",
        "version": "1.4.2",
        "build": "927caff",
        "http_address": "inet[/X.Y.Z.W:9200]",
        "attributes": {
            "aws_availability_zone": "us-east-1b",
            "data": "false",
            "max_local_storage_nodes": "1",
            "master": "false"
        },
        "transport": {
            "bound_address": "inet[/0:0:0:0:0:0:0:0%0:9300]",
            "publish_address": "inet[/X.Y.Z.W:9300]"
        }
    }

[costin]

Can you please turn on logging (see the docs) in particular on the rest package and report back. I'm curious when this discovery occurs. Thanks!

[tzhang101]

I used ip forwarding to workaround this issue. That is I forward the 2 clients to other nodes. It seems working, but if you send me more details on how to enable logging, I can try to generate the log.

Thanks.

Tian

[costin]

This is explained in detail in the reference documentation here. Simply add or modify the log4j.properties used by your environment and add the rest package to it with TRACE level and run the job. You should see the REST/HTTP requests being made right away.
Please remove whatever sensitive information you have and upload them as a gist.
Also what version of Es and Es-Hadoop and Hadoop are you using?

[costin]

Note the latest Beta(4)/master does provide support for client nodes routing if needed.

[juanginzo]

@costin can you point us to where in the docs is the client routing stuff? I'm running on this same problem.