Cache _xpack/usage

[kostasb]

Describe the enhancements:

Cache the output of _xpack/usage Elasticsearch API for a reasonable amount of time (i.e. 10 minutes), to avoid sending this request too often to Elasticsearch. The output could change in case there are changes to Elasticsearch but leave Kibana running, so Kibana could potentially have a stale version of that cached response, for as long as it caches it. Since this call is made from numerous places in Kibana, it makes sense to move this into core platform somewhere, as opposed to i.e. just alerting.

Describe a specific use case for the feature:

A number of views and features in Kibana perform an auth/capabilities check against Elasticsearch's _xpack/usage API.

With a high concurrent number of users in Kibana, these calls can be repeated at a high rate. In case the Elasticsearch master nodes are under load and slow to respond to those calls, Kibana may time out and fail to render the relevant views.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

it makes sense to move this into core platform somewhere

We can't easily introduce a cache mechanism at the ES client level, so this would have to be implemented by the API consumers directly.

Since this call is made from numerous places in Kibana

Seems in opposition with the issue's description, but from a quick grep, seems the only consumer of this API I could find is

kibana/x-pack/plugins/telemetry_collection_xpack/server/telemetry_collection/get_xpack.ts

Lines 18 to 19 in 4584a8b

	export async function getXPackUsage(esClient: ElasticsearchClient) {
	const { body } = await esClient.xpack.usage({ master_timeout: TIMEOUT });

[mshustov]

It's used by Security plugin as well

kibana/x-pack/plugins/security/server/routes/role_mapping/feature_check.ts

Line 92 in 6ecffcc

.request({ method: 'GET', path: '/_xpack/usage' })

cc @elastic/kibana-security as code owner of the check

[DaveCTurner]

FWIW I think there is room for improvement in Elasticsearch too (in addition to any changes in Kibana proposed here). GET _xpack/usage should be fairly lightweight but it seems it isn't in some places, particularly:

• VectorsFeatureSet
• FlattenedFeatureSet
• RuntimeFieldsFeatureSet

I believe we could implement some caching in Elasticsearch for these computations, although I think the only one that's still in play in 7.13 is VectorsFeatureSet anyway, the other two have already been removed.

The cluster in question was also busy serving various get-mappings requests, which uses the same threadpool and can also be quite heavy.

In addition, today in Elasticsearch we don't cancel this work on a client-side timeout (neither get-mappings nor get-xpack-usage) which means that if a backlog builds up then working through it can take a long time and a lot of wasted effort. We have the infrastructure needed to cancel things on client-side timeouts, we just haven't applied it to these particular endpoints yet.

If you think it would help to streamline things in Elasticsearch then please feel free to open issues suggesting:

1. caching so that VectorsFeatureSet#usage does not need to iterate all the mappings on each call, and
2. adding support for client-side cancellability to the get-mappings and get-xpack-usage APIs.

[mikecote]

cc @elastic/kibana-alerting-services

[legrego]

Not sure if this would have any performance implications, but a related request for this endpoint is to allow it to run against the local node instead of forcing the request to get routed to the master node: elastic/elasticsearch#53916

The security plugin's usage that @mshustov pointed out above would ideally use this "local" information if it was made available.

[afharo]

The telemetry use-case calls it once every 24h per sender. #87846 would ensure it's called only once every 24h overall.