Dynamic Index Cache Sizing

[pickypg]

Description

Index-level caching is one of many areas that can be helpfully tuned to improve the performance of Elasticsearch. Ironically, disabling / reducing a cache size is often a solution for improving the performance.

Elasticsearch has tracked hit/miss stats of the query cache, but it does nothing with the data other than inform an interested user about the effectiveness of their index's caching.

It would be interesting if the node would be able to tune the index query cache heuristically based on actual usage (age, frequency, and value). This could likely be combined with more automatic, global tweaks that occur based on the node's data tier.

This could ultimately be used to give GBs of heap back to an individual ES node instead of burning it on cache storage that's not effectively used (or useful).

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jpountz]

I agree with the general observation that Elasticsearch is likely often burning GBs of heap on the filter cache for little benefit. We should more proactively give back heap to the JVM when possible.

In my opinion this is a general Lucene issue, let's move it to Lucene's JIRA?

[pickypg]

https://issues.apache.org/jira/browse/LUCENE-10602

The aspect about tuning it by data tier may still be worthwhile for ES to pick up, such as removing it from the frozen tier.