Set a default value for 'preference' for Kibana ES' client

[dgieselaar]

Per @jpountz's (cautious) recommendation to set a default value for preference to increase shard request cache hit ratio, I would like to suggest that we do this for Kibana's ES client.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

What we're exposing to our API consumers via the elasticsearch core service is a vanilla elasticsearch client. We don't currently have an easy way to alter the queries for some of the client's APIs.

Unless this is critical in some way, I'd really rather wait until this is addressed upstream (or until we have other use cases to wrap the exposed client with a wrapper/proxy that would allow to potentially perform this more easily)

[sorenlouv]

@dgieselaar Can we just do this for APM for the time being?

[dgieselaar]

@sqren we have it for APM already. @pgayvallet I can't really comment on how critical it is. It will increase cache hits by a certain amount when querying shards with 1 or more replicas. I was hoping for an easy fix but if it requires an architectural change than it might not be worth it. But I wouldn't hold your breath for a change on the Elasticsearch side for now, if that is what you were looking for.

[mshustov]

There is a valid concern about making it the default: elastic/elasticsearch#73251 (comment)

This is a tricky question. In practice, we have several benchmarks that suggest that preference
does indeed make response times significantly better by improving cache efficiency.
In theory though, this would prevent adaptive replica selection from routing traffic away
from nodes that are performing poorly, so there might be cases when this would hurt.

So it means we can make it default, but users must have the ability to configure the value to mitigate the potential problems.
The next question is how to make it configurable? We already have data plugin that's meant to be the first-class API for the search operations in Kibana https://github.com/elastic/kibana/blob/e527c3fe297d3e17c6d5a8a24b4ea58a676fc641/src/plugins/data/README.mdx
I propose to define search.preferences in the data plugin config to collocate all the search related config values under a single domain. We are in process of moving elasticsearch.shardTimeout to the data plugin itself #118250 WDYT @elastic/kibana-app-services ?

A downside of the proposal is that the Core cannot leverage configured search.preferences value. Although I don't see us using this API a lot in Core.