CompletionStats need only be recomputed on a refresh

[DaveCTurner]

Computing the completion stats involves walking every field of every segment of every relevant shard, looking for completion fields. By default the seemingly-innocuous GET _stats API does this for every shard in the cluster. I've seen more than a few cases where an external monitoring system is hitting an overly-broad stats API hard enough that the cluster can't keep up. The consequence is that these stats requests pile up in the management threadpool and interfere with the other users of that threadpool.

As far as I can tell, these stats only change on a refresh. In most cases this means they do not change much at all, so I think we can improve the situation by caching these stats between refreshes.

I also note that in #33847 we changed the source of these stats from the external searcher to the internal one. I'm not sure why - external seems more appropriate to me, and would help with the caching since external refreshes may be very infrequent indeed.

Relates:

• completionStats are slow #36773
• https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/7
• https://discuss.elastic.co/t/monitor-tasks-piling-up-in-master-node-6-8-3/212547
• https://discuss.elastic.co/t/concurrent-indices-stats-requests-cause-cluster-to-go-red/161385
• https://discuss.elastic.co/t/high-cpu-usage-while-bulk-indexing/135023/10

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Engine)

[rtkgjacobs]

There are number of external projects (just listing a few we've found with our audits so far) that hit this pinch point by default as an FYI

• https://github.com/vvanholl/elasticsearch-prometheus-exporter

  • This one uses elastics own library https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/action/admin/indices/stats/IndexStats.java (its not clear to me if said lib will default to being 'nice' and not go after suggestion stats? (we avoided its use to avoid having to install, version match plugins per node)

• https://github.com/justwatchcom/elasticsearch_exporter

  • Audit shows when a users uses es.indices, it hits the /_all/_stats/ API - impact high
    

• https://github.com/lmenezes/cerebro

  • Audit shows this is only a minor hit when a user clicks for details on a specific index, otherwise its not polling the greed /stats endpoint (only asks for docs, store) - impact low
  • https://github.com/lmenezes/cerebro/blob/f2ed031fd89258c6ad85d40d846f5ff1f610fe73/app/elastic/HTTPElasticClient.scala

Having the ability to have a default _stats call be far less abusive / greedy r/e above would be ideal. Just tossing in my vote (otherwise for our own deployment, we may be forced to fork various assets and maintain private builds now)

Im sure other tools / fixtures out there can pinch others as it has our own ramping of a large scale set of clusters.

[dnhatn]

+1 to cache the completion stats (especially when we have many fields). Another optimization is to not compute the completion stats if we don't have any suggest field in the mapping.

[rtkgjacobs]

+1 to cache the completion stats (especially when we have many fields). Another optimization is to not compute the completion stats if we don't have any suggest field in the mapping.
We have turned off all things hitting variants of /stats API's and are still finding high management threading / queue backlogs. We confirmed our mappings do not call out or enable suggested fields.

Is ES7 set to always generate completion stats during ingestion ? Can we turn this off? It is possibly a blocker for us to upgrade to ES7 from prior (old) versions in production for us if so... We are finding with our latest tests removing the above fixtures, we are still seeing it stand out in our hot threads. Does x-pack monitoring hit or stimulate the gathering of completion stats?

[DaveCTurner]

Is ES7 set to always generate completion stats during ingestion ?

It's not really anything to do with ingestion, nor is it specific to version 7. By default the completion stats are computed for each stats call and this seems to be true as far back as version 1.7 (I haven't checked further back).

Can we turn this off?

Indeed you can; if you don't need to monitor completion stats so frequently then you should exclude them from these stats requests. I can't comment on how you might configure the third-party products linked above to do this, sorry, but Elasticsearch has had support for selecting specific stats for a long long time.

[rtkkroland]

but Elasticsearch has had support for selecting specific stats for a long long time.

Is this true for /_cluster/stats as well? It doesn't mention it in the Guide https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html and it returns the full set of completion stats.

[DaveCTurner]

@rtkkroland oh dear, yes, I didn't realise we also compute completion stats for the cluster-level API but you're right I think we do:

elasticsearch/server/src/main/java/org/elasticsearch/action/admin/cluster/stats/TransportClusterStatsAction.java

Lines 57 to 59 in 84dbadb

	private static final CommonStatsFlags SHARD_STATS_FLAGS = new CommonStatsFlags(CommonStatsFlags.Flag.Docs, CommonStatsFlags.Flag.Store,
	CommonStatsFlags.Flag.FieldData, CommonStatsFlags.Flag.QueryCache,
	CommonStatsFlags.Flag.Completion, CommonStatsFlags.Flag.Segments);

It doesn't look like there are selectors for this API 😕

[rtkgjacobs]

Some additional follow up / findings on our side. We had Netdata installed on all our cluster nodes and its default configuration hits _cluster/stats - and its default refresh rate seems too frequent. Removing that vastly improved our performance (several orders of magnitude). We have slightly raised the maximum threads of type:management to match our core counts to offset what 30s prometheus scrapes we run. We'll be watching closely balancing these API's use, but passing on a warning for others that might fall into our trap.

Strongly encourage the cost or risks of API's like _cluster/stats and _all/stats etc are derisked where possible. A few of us on this project have more grey hairs now.