LUCENE-10280: Store BKD blocks with continuous ids more efficiently

[gf2121]

For scenes that index is sorted on the field, blocks with continuous ids may be a common situation. In this case we can handle this situation more efficiently. We just need to check

stritylysorted && (docIds[start+count-1] - docids[start] + 1) == count

to check if ids is continuous. If so, we can only write the first id of this block.

[jpountz]

I'll wait before merging to give @iverase a chance to give his opinion on this change but it looks good to me.

[jpountz]

maybe assert that min2max > count?

[gf2121]

+1, fixed

[iverase]

Glad we didn't introduce another implementation of DocIdSetIterator and all there dance around it.

[gf2121]

Hi @jpountz ! Just to remind, maybe we can merge this now? :)

By the way, I found that there is a PR about using readLELongs in BKD apache/lucene-solr#1538. The discussion of this issue has stopped since last Year. This looks promising and I'd like to play with it but i wonder why it stopped, if there are some problems with this idea or if there has already been someone working on this ?

[iverase]

I will merge soon if Adrien does not beat me up.

I worked on the PR about using #readLELongs but never get a meaningful speed up that justify the added complexity. Maybe now that we have little endian codecs might make more sense. I am not planing to continue that work so please feel free to have a go.

[gf2121]

@iverase Thanks for your explanation!

I worked on the PR about using #readLELongs but never get a meaningful speed up that justify the added complexity.

I find that we were trying to use #readLELongs to speed up 24/32 bit situation in the DocIdsWriter, which means the ids in the block are unsorted, typically happening in high cardinarlity fields. I think queries on high cardinality fields spend most of their time on visitDocValues but not readDocIds, so maybe this is the reason that we can not see a obvious gain on E2E took?

My current thoughts are about using readLELongs to speed up the sorted ids situation (means low or medium cardinality fields), whose bottleneck is reading docIds. For sorted arrays, we can compute the delta of the sorted ids and encode/decode them like what we do in StoredFieldsInts.

I raised an ISSUE based on this idea. The benchmark result i post in the issue looks promising. Would you like to help take a look when you have free time? Thanks!