Skip to content

perf(rowids): accelerate dense bitmap decoding - #8715

Merged
Xuanwo merged 6 commits into
lance-format:mainfrom
jiaoew1991:perf/rowid-bitmap-fast-paths
Sep 2, 2026
Merged

perf(rowids): accelerate dense bitmap decoding#8715
Xuanwo merged 6 commits into
lance-format:mainfrom
jiaoew1991:perf/rowid-bitmap-fast-paths

Conversation

@jiaoew1991

@jiaoew1991 jiaoew1991 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Add dense and near-dense bitmap decode paths.

This follows #8713 and targets RangeWithBitmap segments after the sequential cursor has removed repeated prefix scans.

The decoder now:

  • emits a full 0xff byte as one contiguous eight-value range;
  • expands full bytes with at least six set bits as contiguous runs;
  • selects the specialized dense stream once for a single bitmap segment;
  • seeds the cursor with the cardinality used for that decision, avoiding a second bitmap scan;
  • keeps multi-segment sequences on the existing sparse path because one stream-wide decoder cannot safely assume that every segment has the same density.

The sparse cursor and bitmap loop remain separate and source-equivalent to main. This avoids the measurable fallback regression caused by performing adaptive dispatch in every batch.

Bitmap.data and Bitmap.len remain publicly accessible for source compatibility. A proposed popcount cache was removed because direct mutation of the public byte vector could otherwise make the cached cardinality stale. The on-disk encoding remains byte-for-byte unchanged.

Performance

Measured on Linux x86_64 with release-with-debug, 1,000,000 output rows, batch size 1,024, 10 Criterion samples, 1 second warm-up, and 3 seconds measurement. Both binaries used the same benchmark source. Baseline was current main at d57d0fb42; candidate was fbde7d600.

Shape Payload main This PR Change
50% density (holes_2) no 2.243 ms 2.274 ms +0.96% (within Criterion noise threshold)
50% density (holes_2) yes 2.600 ms 2.594 ms -0.22% (no significant change)
~94% density (holes_17) no 2.484 ms 1.600 ms -35.58%
~94% density (holes_17) yes 2.592 ms 1.677 ms -35.14%

Linux perf attributes the dense-shape improvement to the intended decoder change: on main, SegmentCursorState::extend_range accounts for 68.17% of CPU samples; this PR moves that work to SegmentCursorState::extend_dense_range (48.61% of samples) while reducing end-to-end time by 35.58%. For the 50%-density fallback, both main and this PR remain in SegmentCursorState::extend_range (66.83% and 70.13% respectively); no adaptive-dispatch helper appears in the hot path.

Validation

  • cargo test -p lance-table --lib (365 passed)
  • cargo clippy --all --tests --benches -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check
  • full-width-byte regression coverage, including the previous shift-by-8 panic
  • dense sequential cursor coverage across batch and byte boundaries, tail reads, past-end reads, and rewind
  • sparse and multi-segment fallback coverage
  • concurrent multi-batch stable row-ID coverage with deletions
  • unsorted stable row-ID index coverage
  • direct public-byte mutation cardinality coverage
  • public-field source-compatibility coverage through U64Segment::RangeWithBitmap
  • byte-exact serde coverage

Dependency #8713 is merged. This PR targets main directly and contains only the bitmap follow-up.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 24, 2026
@jiaoew1991
jiaoew1991 force-pushed the perf/rowid-bitmap-fast-paths branch from 2e7f052 to 95b4c9c Compare August 24, 2026 12:08
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 24, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 24, 2026
@jiaoew1991
jiaoew1991 changed the base branch from jiaoew/stack-rowid-cursor-base to jiaoew/stack-rowid-cursor-base-v2 August 24, 2026 12:14
@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Aug 24, 2026
@jiaoew1991
jiaoew1991 force-pushed the perf/rowid-bitmap-fast-paths branch from 95b4c9c to c69253b Compare August 24, 2026 13:53
@github-actions github-actions Bot added A-python Python bindings A-deps Dependency updates A-encoding Encoding, IO, file reader/writer labels Aug 24, 2026
@jiaoew1991
jiaoew1991 changed the base branch from jiaoew/stack-rowid-cursor-base-v2 to jiaoew/stack-rowid-cursor-base-v3 August 24, 2026 13:54
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 24, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 24, 2026
@jiaoew1991
jiaoew1991 force-pushed the perf/rowid-bitmap-fast-paths branch from c69253b to 4e6c51e Compare August 24, 2026 14:33
@github-actions github-actions Bot added the A-java Java bindings + JNI label Aug 24, 2026
@jiaoew1991
jiaoew1991 changed the base branch from jiaoew/stack-rowid-cursor-base-v3 to jiaoew/stack-rowid-cursor-base-v4 August 24, 2026 14:33
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 24, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 24, 2026
@jiaoew1991
jiaoew1991 force-pushed the perf/rowid-bitmap-fast-paths branch from 4e6c51e to 470d561 Compare August 25, 2026 01:09
@lance-gatekeeper lance-gatekeeper Bot added K-risk Latest Gatekeeper recommendation includes a non-blocking risk. and removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Aug 26, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 26, 2026
@jiaoew1991
jiaoew1991 force-pushed the perf/rowid-bitmap-fast-paths branch from 19f4049 to e60b7b3 Compare September 2, 2026 09:26
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
@jiaoew1991
jiaoew1991 requested a review from BubbleCal September 2, 2026 11:41
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The latest merge from main leaves the reviewed bitmap patch unchanged relative to its new base, and the incoming changes do not touch the row-ID path. The focused crate suite still passes, so the previously verified dense-decoding speedup, sparse fallback, cursor behavior, public bitmap mutation, and byte-exact serialization remain sound.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
@Xuanwo
Xuanwo merged commit 774e32d into lance-format:main Sep 2, 2026
38 checks passed
Xuanwo added a commit that referenced this pull request Sep 3, 2026
## Summary

Stream dataset-version RLE metadata with a persistent cursor instead of
rebuilding and searching run offsets for every batch.

The change:

- adds `RowDatasetVersionCursor` for adjacent, monotonic batch
traversal;
- caches the active run length and bulk-expands range values;
- lazily builds a run-offset index on the first backward seek, then uses
it for every later non-adjacent seek in either direction;
- precomputes created-at and last-updated-at arrays while polling the
ordered task stream;
- preserves the direct-call fallback and the single-run fast path.

#8713 and #8715 are merged. This branch is rebased on `774e32d67`, and
its diff now contains only the dataset-version work.

## Performance

Linux Criterion A/B, 100,000 rows, batch size 1,024,
`release-with-debug`, pinned to the same CPU. Baseline is `774e32d67`;
this PR is `6f22e566c`.

### Ordered stream

| Version runs | main | This PR | Result |
|---:|---:|---:|---:|
| 1 | 98.325 us | 98.855 us | +0.54% (neutral) |
| 98 | 930.69 us | 112.30 us | 8.29x faster |
| 3,125 | 3.2053 ms | 128.27 us | 24.99x faster |
| 100,000 | 56.033 ms | 517.17 us | 108.34x faster |

### Direct-call fallback

| Version runs | main | This PR | Result |
|---:|---:|---:|---:|
| 1 | 631.81 ns | 634.07 ns | +0.36% (neutral) |
| 98 | 8.983 us | 762.10 ns | 11.79x faster |
| 3,125 | 32.532 us | 5.108 us | 6.37x faster |
| 100,000 | 552.08 us | 142.84 us | 3.87x faster |

This PR makes no single-run speedup claim. It improves robustness for
updated and multi-run fragments.

For attribution, Linux `perf` sampled the 3,125-run ordered-stream case
for five seconds at 997 Hz. On main, 49.72% of samples were in
`version_values_for_selection`, 39.39% in `U64Segment::len`, and 4.96%
in run-offset `Vec` construction (94.07% combined). With this PR,
`U64Segment::len` and offset construction were each below the 0.5%
report threshold; the remaining profile was distributed across cursor
expansion, allocation, Arrow column assembly, and stream scheduling.

## Correctness

The tests cover adjacent ranges, gaps, rewinds, empty runs, non-range
spans, out-of-bounds selections, descending seeks, alternating
far-forward/backward seeks, direct calls, deletions, both version
columns, concurrent batches, and unsorted indices split across multiple
batches.

## Validation

- `cargo test -p lance-table --lib` (372 passed)
- `cargo clippy --all --tests --benches -- -D warnings`
- `cargo fmt --all -- --check`
- `git diff --check`
- Linux `release-with-debug` Criterion A/B and `perf` profiles described
above

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ci CI / build workflows A-deps Dependency updates A-encoding Encoding, IO, file reader/writer A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-python Python bindings K-approved Latest Gatekeeper recommendation permits acceptance. performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants