[elastic] improve hostname resolution

[yannmh]

When the cluster_stats parameter is enabled in the user configuration,
the Datadog Agent's Elasticsearch check attempts to resolve and
attach with the corresponding metrics, the hostname of the different
nodes.

On Amazon Elasticsearch v1.5.2, the node objects
returned by the _cluster/nodes/stats?all=true endpoint returns the
following keys:

root@xxxx:~# cat node_stats.json| jq '.nodes[] | select(.name ==
"${hostname}") | keys
[
  "breakers",
  "fs",
  "indices",
  "jvm",
  "name",
  "os",
  "process",
  "thread_pool",
  "timestamp"
]

where name contains the hostname.

[olivielpeau]

LGTM!

I'm not familiar with Amazon Elasticsearch, do you know if they set the node name to a "sensible" default (since by default on a vanilla ES the default node name is a random Marvel character name).

For reference the node name can be changed in ES' configuration with the node.name config option (see https://www.elastic.co/guide/en/elasticsearch/reference/1.5/setup-configuration.html#node-name)

[yannmh]

do you know if they set the node name to a "sensible" default (since by default on a vanilla ES the default node name is a random Marvel character name).

What do you mean by "sensible" ? Indeed, from what I have seen, the default name is also a random Marvel character name.

[olivielpeau]

By "sensible" I meant "that reflects the hostname or something close/similar"

Basically the check will use that name as the hostname of the metrics that are sent, so my concern is that if the names don't make sense as hostnames we'll have host: tags and hosts in the infrastructure list that use these "weird" names.

It's not a blocker to merge this PR, but if by default the name is still the random Marvel character, I think we should advise users of Amazon Elasticsearch to set the node.name to something that makes sense to them.

[yannmh]

Following our discussion earlier,
name is a unique piece of information for the node. It generally matches the hostname but not always. Thus, let's use it as a tag with the namespace node_name instead of a hostname.

I made the change and will likely have to update the tests too (will see what Travis reports). @olivielpeau can you take a last 👀 on it please ?

[olivielpeau]

Changes look good!

To make the tests pass we should probably set an explicit node.name in conf/elasticsearch.yml (see https://www.elastic.co/guide/en/elasticsearch/reference/1.4/setup-dir-layout.html and https://www.elastic.co/guide/en/elasticsearch/reference/1.4/setup-configuration.html#node-name) and test that the node_name tag is set with that name.