Implement Bits#applyMask in bulk for DenseLiveDocs and SparseLiveDocs - #16593
Open
john-mlika wants to merge 1 commit into
Open
Implement Bits#applyMask in bulk for DenseLiveDocs and SparseLiveDocs#16593john-mlika wants to merge 1 commit into
john-mlika wants to merge 1 commit into
Conversation
john-mlika
force-pushed
the
arc-0-livedocs-applymask
branch
from
September 1, 2026 16:42
6f294fd to
85c0c7e
Compare
john-mlika
force-pushed
the
arc-0-livedocs-applymask
branch
from
September 1, 2026 19:40
85c0c7e to
99e01f0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
since #15413 live docs read from disk are DenseLiveDocs or SparseLiveDocs, and neither overrides Bits#applyMask, so masking a window by live docs went from a word-wise AND (what FixedBits did) to a get() per set bit. every caller pays it on segments with deletions: AcceptDocs#createBitSet, the four bulk scorers and CheckIndex#bitsCardinality. same hole as #16282, other method
dense delegates to FixedBitSet.andRange, bounded on maxDoc, which is what FixedBitSet#applyMask does. sparse gets a new SparseFixedBitSet.andNotRange, the and-not twin of andRange, that clears the non-zero words of the sparse set out of the window word by word. both are one word op per 64 docs at most, whatever the window holds, and both keep FixedBitSet#applyMask's exception for bits past the end of live docs, which Bits#applyMask now documents
what it costs, LiveDocsBenchmark on an AMD Genoa, one mask over a 1M-doc segment with 1% deletions:
on #16588's filtered knn benchmark (200k docs, k=100, 95% filter) a query on a segment with 5% deleted docs takes 2.07 ms on main. with only the applyMask override applied it takes 1.44, with only #16588 it takes 1.23, and with both 0.49, which is where the same query sits on a segment without deletions (0.46). the two PRs are halves of one walk, #16588 builds the accept set in bulk and this masks it in bulk, so either alone leaves half the cost. same shape at 0.1% deleted (2.06 -> 0.49 with both) and at 20% (2.22 -> 0.55)
applies to branch_10x too. Relates to #16282 and #16586