perf(filter): replace bit-at-a-time null bitmap filtering - #11055
Open
Rich-T-kid wants to merge 4 commits into
Open
perf(filter): replace bit-at-a-time null bitmap filtering #11055Rich-T-kid wants to merge 4 commits into
Rich-T-kid wants to merge 4 commits into
Conversation
…evel PEXT The IndexIterator/Indices paths in filter_bits walked the precomputed indices Vec (up to 256 KB at 50% selectivity) to read the null bitmap one byte at a time via get_bit_raw. This caused severe cache pressure against the values buffer being filtered simultaneously. Replace with gather_bits: zip 64-bit chunks from the filter and source bitmaps, apply software PEXT to extract selected bits per chunk. This works directly on the two 8 KB bitmaps, eliminating the indices Vec traversal and reducing source reads from O(count) byte loads to O(filter_len/64) u64 loads. A threshold (count * 64 < filter_len) keeps the original indices path for very sparse selections where the tiny precomputed Vec is cheaper to walk than scanning all filter chunks. Benchmark delta on existing NULLs benchmarks (65536 elements): i32 w NULLs kept 1/2: 149.9 µs -> 38.1 µs (-74%) i32 w NULLs high selectivity: 17.9 µs -> 6.6 µs (-63%) i32 w NULLs low selectivity: 499 ns -> 276 ns (-60%) u8 w NULLs kept 1/2: 159.0 µs -> 63.4 µs (-75%) u8 w NULLs high selectivity: -- -> 2.8 µs (-58%) u8 w NULLs low selectivity: -- -> 234 ns (-35%)
Contributor
Author
|
run benchmark filter_kernels |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/filter-bits-gather-optimization (75281be) to 2078680 (merge-base) diff Run configurationrun benchmark filter_kernels
env:
BENCH_FILTER: "NULL"BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_kernels File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/filter-bits-gather-optimization (75281be) to 2078680 (merge-base) diff Run configurationrun benchmark filter_kernels
env:
BENCH_FILTER: "NULL"CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
commented
Sep 10, 2026
Contributor
Author
|
run benchmark filter_kernels |
1 similar comment
Contributor
Author
|
run benchmark filter_kernels |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/filter-bits-gather-optimization (200598b) to 2078680 (merge-base) diff Run configurationrun benchmark filter_kernels
env:
BENCH_FILTER: "NULL"BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_kernels File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/filter-bits-gather-optimization (200598b) to 2078680 (merge-base) diff Run configurationrun benchmark filter_kernels
env:
BENCH_FILTER: "NULL"BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_kernels File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/filter-bits-gather-optimization (200598b) to 2078680 (merge-base) diff Run configurationrun benchmark filter_kernels
env:
BENCH_FILTER: "NULL"CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/filter-bits-gather-optimization (200598b) to 2078680 (merge-base) diff Run configurationrun benchmark filter_kernels
env:
BENCH_FILTER: "NULL"CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-t-kid/filter-bits-gather-optimization
branch
from
September 11, 2026 19:54
c275451 to
4727e6a
Compare
Contributor
Author
|
run benchmark filter_kernels |
1 similar comment
Contributor
Author
|
run benchmark filter_kernels |
Rich-T-kid
marked this pull request as ready for review
September 11, 2026 19:55
Contributor
Author
|
@sdf-jkl could you take a look 🚀 |
Rich-T-kid
changed the title
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/filter-bits-gather-optimization (4727e6a) to 2078680 (merge-base) diff Run configurationrun benchmark filter_kernels
env:
BENCH_FILTER: "NULL"BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_kernels File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/filter-bits-gather-optimization (4727e6a) to 2078680 (merge-base) diff Run configurationrun benchmark filter_kernels
env:
BENCH_FILTER: "NULL"BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_kernels File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/filter-bits-gather-optimization (4727e6a) to 2078680 (merge-base) diff Run configurationrun benchmark filter_kernels
env:
BENCH_FILTER: "NULL"CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/filter-bits-gather-optimization (4727e6a) to 2078680 (merge-base) diff Run configurationrun benchmark filter_kernels
env:
BENCH_FILTER: "NULL"CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
commented
Sep 11, 2026
Comment on lines
-697
to
-701
| // SAFETY: indices were derived from the filter predicate | ||
| let bits = indices.iter().map(|src_idx| unsafe { | ||
| bit_util::get_bit_raw(buffer.values().as_ptr(), *src_idx + offset) | ||
| }); | ||
| // SAFETY: `Vec::iter()` reports its size correctly |
Contributor
Author
There was a problem hiding this comment.
going to add these safety comments back in
Member
|
I'll take a look this weekend |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
When filtering arrays with a null bitmap, the current code walks one bit at a time
What changes are included in this PR?
Replaces the bit-at-a-time null bitmap loop with a word-level approach: instead of reading one bit per selected row, we load 64 filter bits and 64 source bits at a time, extract only the bits where the filter is set, and pack them directly into the output buffer.
Are these changes tested?
yes, existing test
Are there any user-facing changes?
no