Skip to content

Conversation

@masseyke
Copy link
Member

@masseyke masseyke commented Nov 14, 2025

This fixes the N^2 performance problem described in #97222. In addition to restoring the previous partial fix (#130857), it does the following:

  1. IndicesQueryCache::getStats now accepts a Supplier so that we can only call IndicesQueryCache::getSharedRamSizeForAllShards if it is absolutely needed. This fixes an N^2 performance problem that Improving statsByShard performance when the number of shards is very large #130857 introduced. If a user called TransportIndicesStatsAction but did not request query cache stats, then before Improving statsByShard performance when the number of shards is very large #130857 we did not enter the N^2 loop (it was only entered if a user did request query cache stats). But after Improving statsByShard performance when the number of shards is very large #130857, we had the N^2 performance all the time. This is a pretty big problem for clusters with large shards since this is called very frequently (including every 30 seconds by a background task).
  2. It fixes the N^2 performance in TransportIndicesStatsAction by sharing state across all shardOperation calls on a single node using the new NodeContext feature from Adding NodeContext to TransportBroadcastByNodeAction #138057.

Closes #97222

… to avoid the N^2 performance in TransportIndicesStatsAction if the user did not ask for query cache stats (although it is still there if the user asks for query cache stats). It also avoid O(N) performance in TransportClusterStatsAction if the user did not ask for query cache stats
…performance when a user asks for query cache stats
@masseyke masseyke added >bug :Data Management/Stats Statistics tracking and retrieval APIs v9.3.0 labels Nov 14, 2025
@elasticsearchmachine
Copy link
Collaborator

Hi @masseyke, I've created a changelog YAML for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

>bug :Data Management/Stats Statistics tracking and retrieval APIs v9.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Computing IndicesQueryCache stats is O(N²) in shard count

2 participants