feat(python): add RowIdSequence for constructing row id metadata

[wjones127]

There was no public way to build a stable row id sequence from Python. RowIdMeta could only be constructed via from_json/from_dict, so a caller assembling a transaction externally — a distributed engine computing updates on workers and committing from a driver — had to hand-encode the internal RowIdSequence protobuf to tell Lance that rows in a new fragment already have row ids. Without that, the commit path allocates fresh ids and the rewritten rows silently lose their identity.

This PR adds lance.fragment.RowIdSequence. It builds a sequence from a range, an Arrow integer array, a ChunkedArray, or any iterable of ints, and converts to and from the inline metadata carried on FragmentMetadata.row_id_meta. The commit path already honors a sequence supplied this way, so no changes were needed there.

Duplicate row ids are now rejected rather than silently mis-encoded. U64Segment assumes uniqueness: a repeated value makes SegmentStats::n_holes compute total_slots - count, which underflows. That panics in debug builds, and in release it wraps, steers encoding selection toward RangeWithBitmap, and yields a shorter sequence with a spurious hole — [1, 1, 2] encodes to [1]. The new RowIdSequence::try_from_iter validates before encoding; the existing infallible From conversions are unchanged, so internal callers on hot paths do not pay for the check.

Closes #8353. Addresses the narrowed request in #8317.

Example

Preserving a row's identity in an update assembled outside of Lance — delete the old row, write a replacement fragment, and attach the original row id to it:

from lance.fragment import RowIdSequence, write_fragments

updated_fragment = dataset.get_fragments()[0].delete("id = 2")
(new_fragment,) = write_fragments(pa.table({"id": [2], "v": [99]}), uri)
new_fragment.row_id_meta = RowIdSequence([original_row_id]).to_inline_metadata()

dataset = LanceDataset.commit(
    uri,
    LanceOperation.Update(
        updated_fragments=[updated_fragment],
        new_fragments=[new_fragment],
    ),
    read_version=dataset.version,
)

Reading a fragment's existing sequence back:

sequence = RowIdSequence.from_inline_metadata(fragment.metadata.row_id_meta)
list(sequence)          # [0, 1, 2, 3, 4]
sequence.to_pyarrow()   # uint64 array

Accepted inputs are any of the eight Arrow integer types, a ChunkedArray, a range (a step-of-one range is stored compactly without materializing its values), or any iterable of ints. Nulls, non-integer arrays, negative values, and duplicates each raise a specific error.

Not included

Validation is within-sequence only. Checking that supplied ids are unique across fragments and currently live in the dataset — also requested in #8317 — can only happen at commit time and is left out of this PR.

Documentation for the distributed-write guide will follow in a stacked PR based on this branch.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The typed wrapper addresses the API gap, but this revision still leaves two independent acceptance contracts unresolved: caller-supplied metadata must remain inside the dataset live-ID/allocator invariant, and sequence traversal must stay linear for all supported encodings.

A viable revision can keep typed construction while validating preserved IDs against the candidate live dataset on every commit/rebase (or carry them through an Update-scoped API with that context), and make iteration retain forward traversal state with gapped and multi-segment coverage.

[lance-gatekeeper]

This emits caller-chosen IDs into fragment metadata without any commit-time check that they are existing IDs being relocated. assign_row_ids accepts a complete supplied sequence and leaves next_row_id unchanged, so a fresh supplied ID is reused by the next ordinary append; retained IDs or two replacement fragments can similarly create duplicate live IDs. The result violates the durable global-uniqueness contract and breaks row-ID lookup. Construction-time validation cannot establish dataset liveness: validate the candidate live state during commit and every rebase, or carry preserved IDs through an Update-specific field so allocation and validation are atomic.

Reproducer run on this head

import tempfile
import lance
import pyarrow as pa
from lance import LanceDataset, LanceOperation
from lance.fragment import RowIdSequence, write_fragments

uri = tempfile.mkdtemp() + "/collision.lance"
ds = lance.write_dataset(
    pa.table({"id": [1, 2]}), uri, enable_stable_row_ids=True
)
(new_fragment,) = write_fragments(pa.table({"id": [3]}), uri)
new_fragment.row_id_meta = RowIdSequence([2]).to_inline_metadata()
ds = LanceDataset.commit(
    uri,
    LanceOperation.Update(
        removed_fragment_ids=[], updated_fragments=[],
        new_fragments=[new_fragment], fields_modified=[],
    ),
    read_version=ds.version,
)
ds = lance.write_dataset(pa.table({"id": [4]}), uri, mode="append")
print(ds.to_table(columns=["id"], with_row_id=True)["_rowid"].to_pylist())
ds._take_rows([2])

The IDs printed as [0, 1, 2, 2]; _take_rows then failed with row id index corrupt: stable row id 2 is live in multiple fragments.

[wjones127]

It is up to the caller to guarantee uniqueness. For performance reasons, we don't re-check uniqueness of the row ids on every commit.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The iterator is linear again on this revision, and the focused row-ID suite passes. One acceptance contract remains: the public API says dataset-wide duplicates are rejected, while construction validates only the supplied sequence and commit trusts the caller without advancing the stable-ID allocator.

A caller-owned contract is viable for this advanced commit surface, but it must be explicit where users construct the metadata. Document the provenance, transaction, global-uniqueness, and allocator obligations before exposing it as safe typed metadata.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The public API now states the caller-owned uniqueness and allocator contract, while the implementation keeps sequence-local validation and linear iteration. This resolves the remaining durable row-ID risk without adding a commit-time scan.

[Xuanwo]

Thank you!