Do not route MapReduce reads and writes through non-data nodes

[jasontedor]

The embedded REST client in the MapReduce integration in Elasticsearch Hadoop will by default route through non-data nodes (e.g., dedicated master nodes). This can be problematic as non-data nodes tend to have fewer resources (e.g., network and memory) than data nodes; routing reads using org.elasticsearch.hadoop.mr.EsInputFormat and writes using org.elasticsearch.hadoop.mr.EsOutputFormat through them puts pressure on these resources.

For example, dedicated master nodes usually have smaller heaps than data nodes. Reading and writing from Hadoop MapReduce jobs will cause the network buffers in the Elasticsearch JVM on these nodes to fill up leading to continual GC churn. This churn can lead to GC pauses that cause the node to be temporarily partitioned from the cluster; this is especially problematic when the node is the elected master node.

This can be avoided by disabling es.nodes.discovery and only specifying the data nodes in the cluster in the es.nodes property. However, disabling es.nodes.discovery and managing through es.nodes is not user-friendly, especially for large clusters or clusters with frequent node maintenance.

A more user-friendly approach would be to not route through non-data nodes by default but permit routing through non-data nodes via a configuration option. Since this configuration option will be enabled by default, this is a behavior change from previous versions.

[costin]

Merged in master and once the CI passes, to 2.1.x.
Excellent PR - thanks again for it. Looking forward to the next one ;)

Cheers,