Cross-cluster search support

[epixa]

In 5.3.0, Elasticsearch shipped cross-cluster search, which allows any node to act as a federated client for search/aggregations across multiple clusters without actually joining those clusters. Using cross-cluster search through Kibana is not fully functional at the moment due to limitations in both Kibana and Elasticsearch, but it should be.

The work that we did for tribe node support in Kibana 5.2.0 is mostly applicable to cross-cluster search as well, but there are still a couple of things we need to change in Kibana to function properly in this deployment:

• (Switch to field_caps API #11014) Replace the field_stats call with a call to the new field_caps (new in 5.4) API when creating index patterns and refreshing mappings.
• (Make the field_stats behavior for expanding indices not default for new index patterns #11150) Make the simple query feature the default and recommended option for new index patterns. With this option selected, Kibana will not attempt to intelligently query indices that match the current timepicker range and will instead lean on Elasticsearch for any and all optimizations here. Elasticsearch is working on multiple improvements here, which will make this option performant for most cases:
  • Incremental reduction of query responses First step towards incremental reduction of query responses elasticsearch#23253 (done)
  • Smaller size of terms aggs in incremental reduce Explore possibility of smaller size for terms aggs in incremental reduce elasticsearch#23285 (not necessarily a blocker)
• ([indexPatterns/create] deprecate expand setting #12156) Deprecate the "use field stats" option for new index patterns. We need to hit this pretty hard with some edge case Elasticsearch deployments/searches (tons of shards, for example) to make sure it will be fine for most people and to learn more about what knobs end up impacting Kibana the most with this setting enabled, but assuming the performance is looking good for most cases, we should switch to this as the default and remove the not-recommended text.

All of the above need to be sorted before we can work on support for cross-cluster search in Kibana. When they're finished, there's only one thing that needs to happen in Kibana to make it compatible with the cross-cluster search client out of the box:

Combine field capabilities for all linked clusters. Cross-cluster search only works for search and aggregation requests, so it does combine field capabilities from all clusters. When creating an index pattern, Kibana will need to execute a field capabilities request to each cluster in the federation. There is no current way to retrieve the protocol/host/port for the various clusters, so we're dependent on Elasticsearch for this. elastic/elasticsearch#23925

This will probably only be a couple of requests in most cases, but it could realistically be dozens for very large deployments, so we'll need to throttle this querying as necessary. We need to merge the field capability responses together using the following logic in order to determine the actual capabilities of the multi-cluster search:

• If two fields have different types, there is a conflict and the field can be dropped.
• If any of the fields with the same name is_searchable, the merged field is_searchable.
• If any of the fields with the same name is_aggregatable, the merged field is_aggregatable.

From a Kibana admin perspective, we shouldn't need to require any intervention to work with this deployment. If we can determine what clusters are linked, then we can determine whether we're hitting multiple clusters or not, and if we're hitting multiple clusters, we should disable the same things that we are currently disabling with the tribe support. All of the index pattern/search changes that are happening should work automatically for either a single or multiple cluster deployment.

To perform searches in a cross-cluster search world, we need to specify the name(s) of the cluster(s) to search against. Elasticsearch is going to add wildcard support for this so we can search across all clusters by default: elastic/elasticsearch#23893

[epixa]

/cc @spalger @tylersmalley @jimgoodwin

[epixa]

One thing I need to dig into more is how best to handle the cluster assignment for queries that is necessary with cross-cluster search. Cross-cluster searches should currently specify the cluster name as a prefix to the index name, but Kibana should ideally Just Work™ for querying across all of the clusters. One proposed idea was to add wildcard cluster name support to the cross-cluster API, so we could just make Kibana specify * as the cluster name for all searches, but I'm not clear on where that proposal stands at the moment.

[clintongormley]

One proposed idea was to add wildcard cluster name support to the cross-cluster API, so we could just make Kibana specify * as the cluster name for all searches, but I'm not clear on where that proposal stands at the moment.

The is not currently implemented but I have opened elastic/elasticsearch#23893 - targeting 5.4.0

[clintongormley]

I've opened elastic/elasticsearch#23925 about adding a remote cluster status API and elastic/elasticsearch#23924 about allowing admins to configure a preferred list of HTTP endpoints with each remote cluster.

[bleskes]

@epixa @clintongormley what's the plan around using the remote cluster published http address? Will kibana need to start speaking directly to remote clusters?

[epixa]

The Kibana server needs to hit each remote cluster for field_caps when setting up index patterns.

[bleskes]

The Kibana server needs to hit each remote cluster for field_caps when setting up index patterns.

Thanks. I guess this means we have a long term TODO to be able to proxy these as well, right?

[epixa]

What behavior are you concerned about that warrants a proxy?

[bleskes]

What behavior are you concerned about that warrants a proxy?

To be clear, I meant that ES would send the requests to the other cluster, in the very same way that we "proxy" search requests to remote clusters.

I think it's fine to require that the remote cluster will be accessible to Kibana's server in order to get cross cluster support as soon as possible. Long term it feels like an unnatural restriction - Kibana should be able to just speak to it's single cluster and let it be connected, authenticated etc. to all other clusters. Am I missing something?

[epixa]

@bleskes I do agree that it is awkward from Kibana's perspective, but I'm not sure what the long term plan is here. I'm pretty sure @s1monw felt strongly that this shouldn't be supported directly in Elasticsearch, and I believe his rationale was that it isn't "search" and thus doesn't belong in the cross-cluster search behavior, but I don't want to misrepresent his thoughts here.

[epixa]

FYI: The default search shard limit has been removed elastic/elasticsearch#24012

[s1monw]

@bleskes elasticsearch should definitely NOT proxy anything that can be done from the outside. This discussion comes up from all ends and I am the only one pushing back on this. I am getting tired of it. Elasticsearch can't make any exceptions to the rule otherwise we are going back to tribe and lots of unclear exceptions. If we do field_caps why not node_stats, index settings, explain, $here_goes_yours. We don't do it because it's confusing and elasticsearch should not add syntactic sugar it should provide primitives. That is very important down the road for the success of the product and it's robustness.

[abraxxa]

All questions regarding if Kibana 5.4 can be used with cross cluster search link to this issue which is closed but I'm not sure if that means that it already works and how Kibana and the cross cluster node has to be configured.
Please clarify, thanks!

[epixa]

@abraxxa Kibana does not support cross-cluster search in 5.4. The feature is merged and is currently targeting version 5.5.0.

[epixa]

The PR that introduced cross-cluster search support in Kibana is here: #11114

[abraxxa]

Thanks for clarifying @epixa!