Skip to content

feat(python): expose BlobFile.read_ranges - #8319

Merged
Xuanwo merged 1 commit into
lance-format:mainfrom
keunhong:python-blobfile-read-ranges
Aug 7, 2026
Merged

feat(python): expose BlobFile.read_ranges#8319
Xuanwo merged 1 commit into
lance-format:mainfrom
keunhong:python-blobfile-read-ranges

Conversation

@keunhong

@keunhong keunhong commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

lance.BlobFile gained a positioned read_range(offset, length) (cursor-untouched, GIL-released). Consumers that sample sparse ranges from large blobs — e.g. ML dataloaders fetching 64 spread video frames out of a 1024-frame blob — currently must build caller-side thread pools over read_range to avoid serial object-store round trips. The Rust BlobFile already has the right primitive: read_ranges(&[Range<u64>]), which returns results in input order and lets the ScanScheduler reorder/coalesce/split the physical requests. Only the Python binding is missing. This change exposes it: one vectored call replaces N caller-side threads and gives the scheduler the whole access pattern at once.

Changes

  • pyo3 binding (python/src/dataset/blob.rs): new LanceBlobFile.read_ranges mirroring the read_range pattern — (offset, length) pairs converted to Range<u64> with per-range overflow checks, GIL released across the await via the background executor, PyBytes built after the call returns.
  • Python wrapper (python/python/lance/blob.py): BlobFile.read_ranges(ranges: list[tuple[int, int]]) -> list[bytes] — results in input order; out-of-bounds ranges raise ValueError, matching read_range.
  • Type stubs (__init__.pyi): added read_ranges and the previously missing read_range entry on LanceBlobFile.
  • Tests (python/python/tests/test_blob.py): parity with sequential read_range (empty / empty-ranges / non-monotonic / duplicated+overlapping, parametrized), literal input-order preservation, cursor invariance (including after seek), and out-of-bounds/overflow pinned to ValueError with the same messages the dataset-level read_blob_ranges tests pin.

Purely additive — no Rust-core changes, no new dependencies, no async Python API.

Verification

  • Full python/tests/test_blob.py passes (177 tests).
  • Verified: results identical to sequential read_range on 50 randomized range sets (up to 68 shuffled, duplicated, overlapping ranges over a 4 KB blob), cursor invariant throughout.
  • uv run make format idempotent; uv run make lint (ruff + pyright + rustfmt + clippy -D warnings) clean; all pre-commit hooks pass.

@github-actions github-actions Bot added A-python Python bindings enhancement New feature or request labels Aug 6, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

This exposes the existing Rust multi-range primitive at the right boundary, preserving blob-local bounds, input order, cursor state, and GIL-released I/O while avoiding caller-managed scheduling.

Expose the existing Rust BlobFile::read_ranges as a sync vectored read
on the Python BlobFile. Each range is an (offset, length) pair matching
read_range; results are returned in input order while the underlying
physical requests may be reordered, coalesced, or split by the scan
scheduler. Also add the previously missing read_range entry to the
LanceBlobFile type stub.
@keunhong
keunhong force-pushed the python-blobfile-read-ranges branch from c832b13 to b64ab51 Compare August 7, 2026 01:12

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

This exposes the existing Rust multi-range primitive at the right boundary, preserving blob-local bounds, input order, cursor state, and GIL-released I/O while avoiding caller-managed scheduling.

@Xuanwo
Xuanwo merged commit 118e68c into lance-format:main Aug 7, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants