perf(rowids): stream dataset version runs with a cursor - #8716
Merged
Conversation
This was referenced Aug 24, 2026
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
jiaoew1991
force-pushed
the
perf/row-version-cursor
branch
from
August 24, 2026 12:08
16a33d9 to
450d631
Compare
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
jiaoew1991
changed the base branch from
jiaoew/stack-rowid-cursor-base
to
jiaoew/stack-rowid-cursor-base-v2
August 24, 2026 12:14
lance-gatekeeper
Bot
added
K-approved
jiaoew1991
force-pushed
the
perf/row-version-cursor
branch
from
August 24, 2026 13:53
450d631 to
453f949
Compare
github-actions
Bot
added
A-python
jiaoew1991
changed the base branch from
jiaoew/stack-rowid-cursor-base-v2
to
jiaoew/stack-rowid-cursor-base-v3
August 24, 2026 13:54
lance-gatekeeper
Bot
removed
the
K-approved
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
jiaoew1991
force-pushed
the
perf/row-version-cursor
branch
from
August 24, 2026 14:33
453f949 to
720380c
Compare
jiaoew1991
changed the base branch from
jiaoew/stack-rowid-cursor-base-v3
to
jiaoew/stack-rowid-cursor-base-v4
August 24, 2026 14:33
lance-gatekeeper
Bot
removed
the
K-approved
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
jiaoew1991
force-pushed
the
perf/row-version-cursor
branch
from
August 25, 2026 01:09
720380c to
9c5653f
Compare
github-actions
Bot
added
A-index
jiaoew1991
changed the base branch from
jiaoew/stack-rowid-cursor-base-v4
to
jiaoew/stack-rowid-cursor-base-v5
August 25, 2026 01:09
lance-gatekeeper
Bot
removed
the
K-approved
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
K-approved
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
jiaoew1991
added a commit
that referenced
this pull request
Aug 25, 2026
## Summary Reduce fixed per-batch work in the system-column scan pipeline after the row-ID and version cursors have removed decoder prefix scans. This change: - reads at most 64K stable row IDs ahead and serves adjacent batches as zero-copy Arrow slices; - assembles all requested system columns into one `RecordBatch` instead of repeatedly extending schema/batch objects; - caches structurally equivalent schemas, not only pointer-identical `Arc<Schema>` values; - reuses a full-size zero-column batch template; - skips no-op projections for system-only reads only when schemas are strictly equal; - bounds read-ahead by the actual range selection, including empty and tail reads. The 64K read-ahead bound is about 512 KiB of `u64` values per active fragment. Zero-copy slices can keep that chunk alive until adjacent batches are released. ## Performance 100K-row synthetic scan on top of #8713, #8715, and #8716: - batch size 64: row-ID-only cases improve 29-45%; all-system-column cases improve 53-57%; - batch size 1024, stable isolated all-system cases: | Shape | Payload | Before | After | Reduction | |---|---:|---:|---:|---:| | 50% bitmap | u64 | 544.23 us | 377.30 us | 30.7% | | ~94% bitmap | none | 449.06 us | 291.73 us | 35.0% | | ~94% bitmap | u64 | 481.21 us | 296.43 us | 38.4% | Full-suite batch-size-1024 runs showed pod outliers, so the table reports paired one-case reruns that were stable. The batch-size-64 matrix was stable as a full run. CPU profiling before this change identified `RecordBatch::try_with_column`, `SchemaExt::try_with_column`, schema `Arc` drops, and allocator consolidation as material hotspots. Those repeated per-column/schema-extension hotspots disappear afterward; remaining time is one system-column assembly, row-ID decode, array ownership, and allocator work. ## Validation - `cargo test -p lance-table --lib` (344 passed) - `cargo test -p lance dataset::fragment::tests` (82 passed) - `cargo check -p lance-table --tests --benches` - `cargo clippy -p lance-table --all-targets --no-deps -- -D warnings` - `cargo clippy -p lance --all-targets --no-deps -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` - 32/32 read-only correctness matrix spanning stable-row-ID encodings, boundary offsets, batch sizes 1 through 65536, and row-ID / row-address / combined projections; every array matched an independent oracle Targeted regressions cover mixed payload + all-system projection schema/order, structurally equal schemas with different `Arc`s, stable-row-ID unsorted indices, read-ahead tail/chunk boundaries, empty tasks, and system-only `read_all` / `read_ranges`. Stack dependencies: #8713, #8715, #8716. The PR base is a temporary upstream integration ref containing those three dependencies and will be retargeted to `main` after they merge.
jiaoew1991
force-pushed
the
perf/row-version-cursor
branch
from
August 25, 2026 12:22
9c5653f to
abd74b6
Compare
jiaoew1991
changed the base branch from
jiaoew/stack-rowid-cursor-base-v5
to
jiaoew/stack-rowid-cursor-base-v6
August 25, 2026 12:24
lance-gatekeeper
Bot
removed
the
K-changes
jiaoew1991
changed the base branch from
jiaoew/stack-rowid-cursor-base-v6
to
main
August 25, 2026 12:28
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
BubbleCal
pushed a commit
that referenced
this pull request
Aug 26, 2026
## Summary Reuse a stable row-ID cursor across ordered record-batch tasks and bulk-decode range-backed segments. Stable row IDs represented as `RangeWithBitmap` / `RangeWithHoles` previously rebuilt selection state for every batch. Sequential scans therefore rescanned an ever-growing prefix, approaching quadratic work as batch count increased. This change: - persists `RowIdSequenceCursor` across ordered tasks and caches segment lengths; - adds an exact-capacity contiguous-range path; - adds `SegmentCursorState::extend_range` for bulk expansion of range and bitmap segments; - preserves the direct/random selection fallback and rejects unsorted indices explicitly; - reports truncated stable row-ID metadata as `CorruptFile` instead of panicking or returning a short batch. ## Performance 100K-row synthetic sequential scan, identical Criterion harness: | Batch size | `main` | This PR | Speedup | |---:|---:|---:|---:| | 64 | 17.177 ms | 1.174 ms | 14.6x | | 1024 | 1.868 ms | 236.16 us | 7.9x | CPU profiling on `main` attributed 51.73% to `RowIdSequence::select` and 33.45% to `U64Segment::len`, matching repeated prefix traversal. After this change those hotspots are replaced by `SegmentCursorState::extend_range`; remaining time is fixed allocation/schema work. ## Validation - `cargo test -p lance-table --lib` (336 passed) - `cargo check -p lance-table --tests --benches` - `cargo clippy -p lance-table --all-targets --no-deps -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` This is the root of the row-ID optimization stack and has no dependency on the bitmap or version-cursor follow-ups. Follow-up PRs: - #8715: dense / near-dense bitmap decode paths - #8716: dataset-version RLE cursor - #8747: system-column scan pipeline fixed-cost reduction
jiaoew1991
force-pushed
the
perf/row-version-cursor
branch
from
September 3, 2026 00:51
abd74b6 to
6f22e56
Compare
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
K-approved
lance-gatekeeper
Bot
removed
the
K-approved
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
With #8713 in the base, this patch-equivalent merge preserves the version-column-only change: one-pass adjacent traversal, indexed non-adjacent seeks, and arbitrary selection order across batches.
lance-gatekeeper
Bot
added
the
K-approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stream dataset-version RLE metadata with a persistent cursor instead of rebuilding and searching run offsets for every batch.
The change:
RowDatasetVersionCursorfor adjacent, monotonic batch traversal;#8713 and #8715 are merged. This branch is rebased on
774e32d67, and its diff now contains only the dataset-version work.Performance
Linux Criterion A/B, 100,000 rows, batch size 1,024,
release-with-debug, pinned to the same CPU. Baseline is774e32d67; this PR is6f22e566c.Ordered stream
Direct-call fallback
This PR makes no single-run speedup claim. It improves robustness for updated and multi-run fragments.
For attribution, Linux
perfsampled the 3,125-run ordered-stream case for five seconds at 997 Hz. On main, 49.72% of samples were inversion_values_for_selection, 39.39% inU64Segment::len, and 4.96% in run-offsetVecconstruction (94.07% combined). With this PR,U64Segment::lenand offset construction were each below the 0.5% report threshold; the remaining profile was distributed across cursor expansion, allocation, Arrow column assembly, and stream scheduling.Correctness
The tests cover adjacent ranges, gaps, rewinds, empty runs, non-range spans, out-of-bounds selections, descending seeks, alternating far-forward/backward seeks, direct calls, deletions, both version columns, concurrent batches, and unsorted indices split across multiple batches.
Validation
cargo test -p lance-table --lib(372 passed)cargo clippy --all --tests --benches -- -D warningscargo fmt --all -- --checkgit diff --checkrelease-with-debugCriterion A/B andperfprofiles described above