Error writing to ES from Spark via REST

[kumarlalapet]

OS: : Red Hat Enterprise Linux Server release 6.8 (Santiago)
JVM : Open JDK 1.8
Hadoop/Spark: 1.6.1
ES-Hadoop : 5.0.0
ES : 5.2.0

Three node es cluster is behind a load balancer. SSL communication to the load balancer from loadbalancer to es its NOT encrypted.

Loaded the cert file in the trust store in the jvm cacerts file. Provided this as an argument to the executors via the spark-submit command using -Djavax.net.ssl.trustStore option. Provided the following in the code

                ImmutableMap.Builder builder = configBuilder.put(ConfigurationOptions.ES_NODES,esNodes) // es nodes
                        .put(ConfigurationOptions.ES_PORT,esPort) //9200
                        .put(ConfigurationOptions.ES_NET_USE_SSL,"true")
                        .put(ConfigurationOptions.ES_NET_PROXY_HTTPS_HOST,esProxy) //load balancer 
                        .put(ConfigurationOptions.ES_NET_PROXY_HTTPS_PORT,esProxyPort); //LB port

However getting error as below

org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed

I tried various other options. But nothing worked.

NOTE

1. when I call the url via the browser it works
2. wrote a simple java program using elastic rest client that also works in the same jvm

Need help. Thank you.

[jbaiera]

@kumarlalapet Could you increase the logging level of ES-Hadoop to trace and post the job logs here? Also, ES-Hadoop 5.0 may not be fully forwards compatible with ES 5.2.0. I would make sure to keep your ES-Hadoop library version above your ES version to ensure maximum compatibility.

[kumarlalapet]

@jbaiera Thank you for your response. Will do the same. In the meantime, it seems to work with the following settings

 ImmutableMap.Builder builder = configBuilder.put(ConfigurationOptions.ES_NODES,esNodes) // es nodes
                        .put(ConfigurationOptions.ES_PORT,esPort) //9200
                        .put(ConfigurationOptions.ES_NET_USE_SSL,"false")
                        .put(ConfigurationOptions.ES_NET_PROXY_HTTPS_HOST,esProxy) //load balancer 
                        .put(ConfigurationOptions.ES_NET_PROXY_HTTPS_PORT,esProxyPort); //LB port

The only difference I set the use ssl to false. Is this the correct configuration? Like I mentioned the ES nodes are front ended with a Load Balancer, will the communication from Spark to LB be encrypted with the above setting even when use ssl is set to false?

[kumarlalapet]

@jbaiera This is what I found so far

Like I mentioned I have 3 nodes no SSL for these. Front ended by a load balancer which has SSL enabled.

Scenario - provides load balancer as the es node, and es.net.ssl as true. Following is the trace

[DEBUG] 2017-02-24 12:48:36,551 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransportFactory create - Creating new CommonsHttpTransport
[DEBUG] 2017-02-24 12:48:36,567 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport setupSSLIfNeeded - SSL Connection enabled
[INFO] 2017-02-24 12:48:36,581 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport addHttpAuth - Using detected HTTP Auth credentials...
[TRACE] 2017-02-24 12:48:36,582 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport <init> - Opening HTTP transport to xx.xx.xx.xxx-lb-ip
[TRACE] 2017-02-24 12:48:36,590 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport execute - Tx [GET]@[xx.xx.xx.xxx-lb-ip][] w/ payload [null]
[TRACE] 2017-02-24 12:48:42,295 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport execute - Rx @[xx.xx.xx.xxx-lb-ip] [200-OK]

As you could see above its connecting to LB with SSL and with the auth provided. 200 - OK. returned the cluster name etc. Now node discovery

[DEBUG] 2017-02-24 12:48:42,325 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransportFactory create - Creating new CommonsHttpTransport
[DEBUG] 2017-02-24 12:48:42,325 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport setupSSLIfNeeded - SSL Connection enabled
[INFO] 2017-02-24 12:48:42,325 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport addHttpAuth - Using detected HTTP Auth credentials...
[TRACE] 2017-02-24 12:48:42,325 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport <init> - Opening HTTP transport to xx.xx.xx.xxx-lb-ip
[TRACE] 2017-02-24 12:48:42,326 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport execute - Tx [GET]@[xx.xx.xx.xxx-lb-ip][_nodes/http] w/ payload [null]
[TRACE] 2017-02-24 12:48:47,995 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport execute - Rx @[xx.xx.xx.xxx-lb-ip] [200-OK] [{"_nodes":....... }]
[TRACE] 2017-02-24 12:48:48,002 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport close - Closing HTTP transport to xx.xx.xx.xxx-lb-ip

Got the nodes list as well. Now below trying to connect to one of the node via SSL.. however like I mentioned above only to LB its SSL not to the nodes... and I get the error now. How do I over come this.

[TRACE] 2017-02-24 12:48:48,003 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport <init> - Opening HTTP transport to xx.xx.xx.-es-node-ip
[TRACE] 2017-02-24 12:48:48,003 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport execute - Tx [GET]@[xx.xx.xx.-es-node-ip][_nodes/http] w/ payload [null]
[INFO] 2017-02-24 12:48:48,013 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry - I/O exception (javax.net.ssl.SSLException) caught when processing request: Unrecognized SSL message, plaintext connection?

Looking at the above I changed to the following setting

Scenario - one of the es ip as es node setting, load balancer as https proxy, ssl to true and provided the credentials for https proxy as well.

[DEBUG] 2017-02-24 12:38:36,751 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransportFactory create - Creating new CommonsHttpTransport
[DEBUG] 2017-02-24 12:38:36,767 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport setupSSLIfNeeded - SSL Connection enabled
[DEBUG] 2017-02-24 12:38:36,774 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport setupHttpOrHttpsProxy - Using authenticated HTTPS proxy [xx.xx.-lb-ip]
[INFO] 2017-02-24 12:38:36,783 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport addHttpAuth - Using detected HTTP Auth credentials...
[TRACE] 2017-02-24 12:38:36,783 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport <init> - Opening HTTP transport to xx.xx.-es-node-ip
[TRACE] 2017-02-24 12:38:36,791 org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport execute - Tx [HTTPS proxy xx.xx.-lb-ip][GET]@[xx.xx.-es-node-ip][] w/ payload [null]
[INFO] 2017-02-24 12:38:36,856 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry - I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server xx.xx.-es-node-ip failed to respond
[INFO] 2017-02-24 12:38:36,856 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry - Retrying request

Whats the correct setting and how to make it work. Again the three es nodes are behind a load balancer. No ssl between and to es nodes. However ssl to load balancer.

Key requirement all communication should be via SSL from the client. In this case Spark.

[kumarlalapet]

@jbaiera I got this information form the doc and seems to be pertaining to my issue here

https://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html

es.nodes.wan.only (default false)
Whether the connector is used against an Elasticsearch instance in a cloud/restricted environment over the WAN, such as Amazon Web Services. In this mode, the connector disables discovery and only connects through the declared es.nodes during all operations, including reads and writes. Note that in this mode, performance is highly affected.

So by default the API is doing node discovery via the LB (SSL is fine here as its SSL based) and once it got the node details. Its trying to connect to one of the node via SSL where as the node is just plain HTTP based. Thats why getting the error.

So I changed the setting to the following es.node=lb, es.ssl=true and wanonly=true. Now all the calls are via the LB only and its SSL based and no node discovery. Eeverything worked.

My follow up question is. whats the performance penalty I incur when I do this versus the node discovery.

[jbaiera]

@kumarlalapet You literally just beat me to the punch! re: the es.nodes.wan.only setting.

As for performance penalty: In this case you'll be mostly limited by the available network resources of your load balancer. Normal execution of the connector would allow tasks to communicate with each of the nodes separately and directly, but in this case all traffic will be routed through the load balancer for all Spark tasks that execute. In some workflows this may mean that you'll have to scale back the task parallelism to keep the load balancer happy. You may also see a degradation in job speed due to the extra network IO time of the load balancer forwarding requests. When the wan only setting is enabled, the connector attempts to interact with the cluster through the provided node addresses only, treating that list of addresses as if it were the entire list of available nodes. This disables other optimizations such as avoiding writing to nodes that aren't primaries, or reading from nodes that may not have the target shards locally. You'll have to weigh those performance concerns against your original architectural needs for the load balancer. In practice we find that these hits to performance vary depending on your use case and processing workload.

[jbaiera]

Considering the issue is resolved with configuration changes, I'm going to close this issue. If you have any other questions, feel free to ask.

[kumarlalapet]

@jbaiera thank you for the explanation.

I have a follow on question. Say I dont want the wan only setting to true. And want that SSL as well. So do I need to make the nodes in ES to be SSL enabled?

So that means communication to LB is SSL. And from LB to nodes is also SSL. But not the intra node communication. How can I make that happen?

[kumarlalapet]

@jbaiera and another question. Is there a difference in behavior between the es-hadoop-rest vs the java-rest-client. The reason I ask is, I do not see the wan only setting in the RestClientBuilder from the following package

        <groupId>org.elasticsearch.client</groupId>
        <artifactId>rest</artifactId>

[jbaiera]

@kumarlalapet When using a proxy for communicating with Elasticsearch, you'll need to make a decision on how you want that proxy to behave: Either the proxy is an open proxy, which your hadoop tasks are using in order to forward tasks to nodes they are already aware of, or it's a reverse proxy, which your hadoop tasks are using as a surrogate for the entire Elasticsearch cluster.

If you wish to use an open proxy layout (hadoop tasks are aware of all elasticsearch nodes, it just communicates to them THROUGH the proxy as a middle point) you will need to set the proxy host and proxy ports to use your proxy information, as well as set the starting nodes to be the addresses of one of your actual elasticsearch nodes. If the communication needs to be in SSL, make sure that the open proxy is configured as an SSL termination proxy, otherwise it will just forward the encrypted request on to ES, which will puke. Your second scenario above should be the correct settings for this, it's just that for some reason the response is coming back empty from the proxy. I would check the logs in on your cluster, as well as on the proxy itself to see if something is getting hung up.

If you wish to use your proxy as a reverse proxy, then you need to configure the initial nodes to be the proxy address only, and set wan only to be true. This tells the connector to only communicate with the proxy, and to let the proxy do all the traffic management as it may not have any other route to communicate with the nodes/services behind the proxy. In your first scenario above it seems like you're trying to do this (setting the initial node to be the proxy-lb, but not setting wan only). When the connector starts up, it discovers all the nodes behind the load balancer and attempts to communicate with them using SSL, which then goes on to break since they are not set up with SSL. If you enable the wan-only option, the connector is instructed to do no discovery of the nodes behind the initial connection, and to use that connection as if it were the only node in the target cluster, relying on the proxy to correctly forward the traffic to the backing nodes, which will then forward the requests to the correct shards, etc... etc...

RE: Differences between the java-rest-client: ES Hadoop implements it's own rest client logic in order to maintain parity with the dependencies inside of Hadoop and Spark. The ES-Hadoop client code predates the official java-rest-client. The official rest client also depends on different versions of http libraries that may cause classpath conflicts in Hadoop and Spark jobs.

[kumarlalapet]

@jbaiera This is a great explanation. Thank you so much.

If the communication needs to be in SSL, make sure that the open proxy is configured as an SSL termination proxy, otherwise it will just forward the encrypted request on to ES, which will puke.

This is the error I was getting before. In this case either the nodes should be SSL or the proxy should be a SSL termination proxy. Can you provide examples for both the setup? I mean what settings to be done in the code.

Thank you.

[kumarlalapet]

@jbaiera

The ES-Hadoop client code predates the official java-rest-client. The official rest client also depends on different versions of http libraries that may cause classpath conflicts in Hadoop and Spark jobs.

The java-rest-client - does it do the three calls like the hadoop rest client? The ES server check, node discovery and connecting to the node? Or it just talks to the proxy and thats it? Is there a wanonly setting in the java-rest-client ?

[jbaiera]

@kumarlalapet regarding SSL configuration on the Elasticsearch side, this guide is all that I'm aware of. Configuring SSL termination on your proxy depends entirely on what technology you are using for your proxy.

Regarding the java-rest-client: I don't believe it contains an option for a WAN-Only mode. The official client includes a "sniffer" module that will run a background thread to perform service discovery. ES-Hadoop does two things when wan-only mode is toggled on: 1) It disables discovery of other nodes and 2) disables the specific targeting of nodes that have certain roles (preferring writing to data nodes and ignoring master nodes, etc). If you wanted to emulate wan-only mode on the java-rest-client you could just leave the sniffer off and only connect to your configured nodes. From my understanding, there is no node preference code available at this time in the sniffer module (though there may be in the future), and by leaving the sniffer off the client will do no discovery operations.