[META] Concurrent Searching

[reta]

Is your feature request related to a problem? Please describe.
At least since Apache Lucene 6.x, there is a new experimental low-level API which allows to parallelize execution of the search across segments [3]. As of latest Apache Lucene 8.10.1, the API is still marked as experimental (see please [1]). The community feedback on this feature is looking positive so far (see please [2]), there are high chances that for certain kind of indices parallelizing the search over segments could bring performance benefits.

[1] https://lucene.apache.org/core/8_10_1/core/org/apache/lucene/search/IndexSearcher.html#search-org.apache.lucene.search.Query-org.apache.lucene.search.CollectorManager-
[2] https://engineeringblog.yelp.com/2021/09/nrtsearch-yelps-fast-scalable-and-cost-effective-search-engine.html
[3] https://blog.mikemccandless.com/2019/10/concurrent-query-execution-in-apache.html

Describe the solution you'd like
Support the concurrent search over Apache Lucene segments

Describe alternatives you've considered
N/A

Additional context

• Concurrent Searching #1286 - the concurrent-search sandbox plugin is available
• Concurrent Searching: Support Aggregations #2585
• Concurrent Searching: Support Early Termination #2586
• [FEATURE] Documentation - Document 'concurrent-search' sandbox plugin in OpenSearch 2.1.0+ documentation-website#482
• Concurrent Segment Search Overview and Aggs HLD #6798
• Add support using sandbox functionality in official releases #3677
• Concurrent Searching: add merge policy that evens-out segment size #7160
• [Discuss] Move concurrent-search out of sandbox plugin to core under feature flag #6859
• Concurrent Searching: specialized thread pool implementation #7583

[msfroh]

@reta -- out of curiosity, is there any plan to link concurrent search with a merge policy that evens-out segment sizes?

On Amazon Product Search, we used a merge policy that dynamically adjusts the max segment size (to something like min(5GB, max(1GB, totalIndexSize/5)), IIRC) combined with a merge-on-commit setting that would merge all segments less than some threshold (something like 100MB). Basically, "lower the ceiling and raise the floor".

(I talked about this around minute 14:00 of https://www.youtube.com/watch?v=UwclHSeE_B8. Sorry for the shameless plug. 😁 )

[reta]

@msfroh this is great idea I think, I remember we have discussed that at OpenSearchCon as well, will follow up with the issue, thank you!

[hdhalter]

Hi @yigithub , if anything here is related to 2.10, please create a doc issue or PR for the update. Thanks!