Utilize AcceptDocs interface for lazy evaluation of filters for `*_flat` vector indices

[benwtrent]

Description

We currently have higher level logic to lazily evaluate the filter for our flat codecs. However, with the new AcceptDocs interface, it seems to me we should be able to delegate that down to our flat formats. This way they can simply utilize the AcceptDocs iterator when scoring instead of fully realizing the bitset (which can be truly costly).

I foresee this being split between multiple PRs:

• First, utilizing accept docs for lazy iteration within our custom flat formats.
• Second, removing the custom logic and verifying with benchmarks that performance is still acceptable. (this will require running with the https://github.com/elastic/rally-tracks/tree/master/random_vector track I think)

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[0xShubhamSolanki]

Hi @benwtrent,
I'd like to work on this issue. I've reviewed the description and understand that the goal is to replace the eager BitSet-based filter evaluation in *_flat vector indices with lazy evaluation using the AcceptDocs (DocIdSetIterator) interface to improve memory efficiency and performance.

I plan to:

1.Update the flat vector scoring logic to accept and iterate over a DocIdSetIterator instead of a full Bits instance.
2.Remove the legacy eager filter materialization once the new path is verified.
3.Validate correctness and performance using the random_vector Rally track.

Could you please assign this issue to me? Thanks!

[benwtrent]

Your plan seems good to me.

[madalynbartman]

@benwtrent I gave the first bullet (utilizing accept docs for lazy iteration within our custom flat formats) a shot in #139145. Please take a look at it and let me know if I need to adjust my approach. Thanks!