branch-4.1: [fix](search) Replace ExcludeScorer with null-bitmap-aware version for MUST_NOT to handle NULL rows #61200#61323
Merged
yiguolei merged 1 commit intoapache:branch-4.1from Mar 14, 2026
Conversation
…r MUST_NOT to handle NULL rows (apache#61200) Cherry-pick of apache#61200 to branch-4.0. search('NOT msg:omega') incorrectly included NULL rows because ExcludeScorer did not implement has_null_bitmap()/get_null_bitmap(). Enhanced ExcludeScorer now accepts a pre-collected exclude_null bitmap and checks it during advance()/seek() before the original lazy is_within() exclusion. Key changes: - ExcludeScorer: add _exclude_null/_null_bitmap members, override has_null_bitmap()/get_null_bitmap(), null check in constructor/advance/seek - OccurBooleanWeight: add build_exclude_opt() to collect null bitmaps from MUST_NOT scorers and union them; pass exclude_null to make_exclude() - Plumb binding_keys from function_search.cpp through OccurBooleanQueryBuilder -> OccurBooleanQuery -> OccurBooleanWeight for proper null bitmap resolution - Add DCHECK for _binding_keys size validation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #61200 to branch-4.1.
What problem does this PR solve?
Issue Number: close #DORIS-24681
search('NOT msg:omega')incorrectly includes NULL rows in the result set, whileNOT search('msg:omega')correctly excludes them.Root cause:
ExcludeScorerdoes not implementhas_null_bitmap()/get_null_bitmap(), inheriting theScorerbase class defaults that always returnfalse/nullptr. This caused NULL documents to be treated as TRUE (matching) rather than NULL, violating SQL three-valued logic whereNOT(NULL) = NULL.Fix: Enhance
ExcludeScorerwith null bitmap support:exclude_nullbitmap (read from index at O(1) cost)advance()/seek()before the original lazyis_within()exclusionhas_null_bitmap()/get_null_bitmap()to propagate null infobinding_keysfromfunction_search.cppthrough OccurBooleanQueryBuilder → OccurBooleanQuery → OccurBooleanWeight for proper null bitmap resolutionRelease note
Fix search('NOT field:value') incorrectly including NULL rows by using null-bitmap-aware ExcludeScorer.
Check List (For Author)
Check List (For Reviewer who merge this PR)