GITHUB#12370: early-terminate string sort when field is missing from the whole index#16411
Open
serhiy-bzhezytskyy wants to merge 1 commit into
Open
Conversation
…the whole index
When sorting on a string (SortedSet/SortedDoc) field that does not exist
anywhere in the index and the sort has no tie breaker, all documents tie on
the missing value. Once the top-N queue is full, a further missing value can
no longer compete, so the remaining documents should be skipped instead of
fully collected.
TermOrdValComparator only recognized this on the maxOrd side; the ascending
minOrd branch hard-coded minOrd=-1 ("missing still competitive"), which tripped
the guard that disables skipping. Mirror the singleSort recognition onto the
minOrd side, gated on bottomValue==null (a genuine missing value in the queue),
which is stronger than bottomOrd==missingOrd and so does not misfire in the
sort-missing-first mixed-segment case.
Signed-off-by: Serhiy Bzhezytskyy <me@serhiy-bzhezytskyy.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #12370.
Description
Sorting on a string field that is missing from the whole index currently scans every document instead of early-terminating.
When the sort field doesn't exist anywhere in the segment, every document ties on the "missing" value. With no tie-breaker, once the top-N priority queue is full, all remaining documents are non-competitive and can be skipped — but
TermOrdValComparatordidn't recognize this case, so it visited every doc.Fix
TermOrdValComparatornow mirrors the existingsingleSortcompetitive-iterator recognition to theminOrdpath, gated onbottomValue == null(i.e. the field is missing from the segment). This lets the comparator signal non-competitiveness and skip once the queue is full, exactly as it already does for present fields.Correctness is unchanged — results are identical; this is purely an optimization (so it's filed under Optimizations in CHANGES.txt).
Tests
Added a regression test to
TestSortOptimizationthat asserts early termination happens (non-vacuous — it fails without the fix)../gradlew :lucene:core:test --tests TestSortOptimization→ 29 tests, all green on current main.