Cross-cluster search remote cluster status API

[clintongormley]

With cross-cluster search configured, Kibana needs to be able to:

• discover the names of the remote clusters
• discover the remote cluster's http.publish_address
• check if remote clusters are contactable

This is made more complicated by the fact that cluster settings can be persistent, transient, or default (configured in the elasticsearch.yml file).

Let's add a remote cluster status API like the following:

GET _remote/status

which could return something like:

{
  "cluster_one": {
    "seeds": [
      "10.0.0.1:9300"
    ],
    "http_addresses": [
      "10.0.0.1:9200",
      "10.0.0.1:9201",
      "10.0.0.1:9202"
    ],
    "connected": true,
    "latency_in_ms": 4,
    "connections_per_cluster": 3,
    "initial_connect_timeout": "30s",
    "level": "cluster|node"
  },
  "cluster_two": {...}
}

The http_addresses could be pre-configured (see #23924) or be sniffed from the remote cluster's node info. (the same questions about HTTPS apply).

The level key would indicate whether this remote cluster has been configured via the cluster settings or locally on the node.

[clintongormley]

/cc @s1monw

[clintongormley]

I chatted to @s1monw about how we should handle http vs https. Cross-cluster search will only work if all or none of the clusters are using TLS. Of course, it is possible that TLS is enabled on the transport layer but not on the HTTP layer.

Outcome of the discussion is:

• if kibana is using credentials, then assume https (as we should never pass creds in the open)
• if not using credentials, Kibana can issue two requests in parallel and cancel the second one once the first returns

Also, instead of configuring HTTP addresses in the elasticsearch cluster state, we'll return a list of 3 addresses sniffed from the publish address in the nodes info of the remote cluster. If the clusters are set up with search.remote.node.attr, the list of http addresses will be pulled from those nodes which have the attribute. (see https://www.elastic.co/guide/en/elasticsearch/reference/5.3/modules-cross-cluster-search.html#_cross_cluster_search_settings)