Group together search requests that hit the same shard in the search queue

[jpountz]

On dense tiers like warm and cold the page cache can only cover a tiny fraction of the index, while disk access is expensive as these dense tiers often leverage spinning disks.

Could we reorder items in the search queue in order to group together shard requests that would hit the same shard in order to maximize the hit ratio of the page cache?

This is probably something that we should avoid doing on the content tier where this could trade latency for throughput in non-desirable ways?

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jpountz]

One concrete implementation idea to make this work would be to switch the search queue to a PriorityQueue, that would be ordered by the time when the request entered the queue rounded to 10s, then the shard identifier (index UUID + shard number)?

Using the timestamp first helps make sure that some shard requests would not get indefinitely delayed if new shard requests with a lower shard identifier kept coming in. The rounding granularity could possibly depend on the tier, e.g. 10s on warm/cold and 1m on frozen?