No public API to preserve stable row IDs when assembling an Update transaction externally

[justinrmiller]

Environment: pylance 9.0.0-beta.21 (behavior verified empirically); source citations from v9.1.0-beta.2 (423c1943).

LanceDataset.update() preserves stable row IDs when it rewrites a row. The same edit assembled externally — delete the old row, write a replacement fragment, commit LanceOperation.Update — cannot, because there is no public way to tell Lance that the appended rows carry pre-existing row IDs. The rewritten row silently receives a fresh ID.

This blocks distributed engines that compute updated values on workers and assemble the transaction on a driver.

Repro

import tempfile
import lance
import pyarrow as pa
from lance.fragment import write_fragments

def ids(ds):
    t = ds.to_table(columns=["id"], with_row_id=True)
    return dict(zip(t["id"].to_pylist(), t["_rowid"].to_pylist()))

# (a) built-in update preserves stable row ids
uri_a = tempfile.mkdtemp() + "/a.lance"
ds = lance.write_dataset(pa.table({"id": [1, 2, 3, 4], "v": [10, 20, 30, 40]}),
                         uri_a, max_rows_per_file=2, enable_stable_row_ids=True)
before = ids(ds)
ds.update({"v": "99"}, where="id = 2")
ds = lance.dataset(uri_a)
print(ids(ds) == before)                                            # True
print(ds.read_transaction(ds.version).operation.update_mode)        # rewrite_rows

# (b) the same edit assembled externally loses the row id
uri_b = tempfile.mkdtemp() + "/b.lance"
ds = lance.write_dataset(pa.table({"id": [1, 2, 3, 4], "v": [10, 20, 30, 40]}),
                         uri_b, max_rows_per_file=2, enable_stable_row_ids=True)
before = ids(ds)
updated = ds.get_fragments()[0].delete("id = 2")
new = write_fragments(pa.table({"id": [2], "v": [99]}), uri_b)
op = lance.LanceOperation.Update(
    removed_fragment_ids=[], updated_fragments=[updated],
    new_fragments=list(new), fields_modified=[],
)
lance.LanceDataset.commit(uri_b, op, read_version=ds.version)
print(ids(lance.dataset(uri_b)) == before)                          # False

Output:

(a) built-in update, ids preserved: True
    update_mode recorded: rewrite_rows
(b) externally assembled update, ids preserved: False
    row holding id=2 was rowid 1 -> now 4

Note that (a) and (b) are the same shape — update_mode is rewrite_rows, i.e. delete-in-place plus append. The built-in path preserves the ID because it carries the original sequence through internally; an external caller has no way to express that.

Why there is no workaround today

• write_fragments exposes only enable_stable_row_ids: bool — no way to supply IDs. (Per write_fragments(enable_stable_row_ids=True) silently produces fragments without row-id metadata, rejected on commit #7702 that flag is a no-op there anyway, and commit-time assignment is the sound design for newly inserted rows, since sequences come from the manifest's single next_row_id counter. This request is the different case: rows that already have IDs and are being rewritten.)
• FragmentMetadata accepts row_id_meta, but RowIdMeta has no constructor — RowIdMeta(...) raises TypeError: cannot create 'lance.fragment.RowIdMeta' instances. The only way in is from_json/from_dict, i.e. hand-authoring the internal RowIdSequence protobuf (protos/rowids.proto). Consumers pin a version floor, so a change to that internal encoding would silently mis-assign row IDs — the exact invariant at stake.
• UpdateMode::RewriteColumns does preserve IDs, since fragment identity is retained. But it rewrites the full column within every touched fragment, so cost is O(rows in touched fragments) rather than O(matched rows). For a low-selectivity update (say 10 matched rows across three 1M-row fragments) that is a ~10⁵× write amplification, which defeats the point of a sparse update.
• Relatedly, the Python→Rust conversion for Update hardcodes updated_fragment_offsets: None and inserted_rows_filter: None (python/src/transaction.rs), so the partial RewriteColumns form — documented in transaction.rs as "Used with stable row IDs so build_manifest can refresh row-level version metadata only for rows that were rewritten" — is not expressible from Python either.

Proposed fix

In preference order:

1. Let the caller supply preserved IDs on LanceOperation.Update — e.g. a field parallel to new_fragments carrying, per new fragment, the pre-existing row IDs its rows correspond to. The Update arm would consume those in place of allocating from next_row_id. This keeps the sequence encoding private and is the minimal surface that unblocks the case.
2. A public constructor for row-id sequences — RowIdMeta.from_ids([...]) or an equivalent builder — so FragmentMetadata(..., row_id_meta=...) is expressible without hand-encoding protobuf. More general, but exposes the encoding as API.
3. Expose updated_fragment_offsets (and inserted_rows_filter) through the Python Update conversion, so partial RewriteColumns is reachable. Useful independently of 1 and 2.

Validation for either of 1 or 2 should reject supplied IDs that are not currently live in the dataset, and reject duplicates, so a malformed external transaction fails at commit rather than corrupting the ID space.

Relationship to #7702

#7702 covers write_fragments(enable_stable_row_ids=True) being a silent no-op and the Merge arm skipping ID/fragment-ID assignment. This issue is about the Update arm and is not fixed by that one: even with write_fragments behaving correctly, there is still no way to say "these rows keep the IDs they already had." Both trace to the same root — row-ID assignment happens at commit and external callers cannot participate — so they may be worth addressing together.

Impact

Found while making a distributed sparse-update path preserve row identity. Downstream, consumers key on _rowid to join derived tables back to source rows; when an update silently reassigns IDs, those consumers treat the rewritten rows as new (duplicating derived rows) while the rows keyed by the old IDs are orphaned. The only currently-available correct option is the full column rewrite, which erases the performance advantage that motivated the sparse path.

[Xuanwo]

Based on my observation of #8318, this is more like a feature request than a quick bug to fix.

I would suggest engaging @wjones127 to figure out the overall design first.

[westonpace]

So I think this would mean that the call to delete returns the list of deleted row ids. Then you would need to be able to attach those as part of the write_fragments call.

Currently this is how it works internally. The newly created fragments have the row id sequence set as the row_id_sequence field of the DataFragment. As long as that is done then the commit should behave appropriately.

We do not change or modify the row id sequence of the deleted fragment. Simply adding a deletion vector is sufficient.

[justinrmiller]

Thanks — that's exactly what I needed, and it works. Setting the sequence on the new fragment is sufficient, confirmed end to end on pylance 9.0.0-beta.21: taking repro (b) above and attaching row_id_meta to the replacement fragment preserves the row's identity, with the deletion vector alone on the original fragment as you describe.

updated = ds.get_fragments()[0].delete("id = 2")
f = list(write_fragments(pa.table({"id": [2], "v": [99]}), uri))[0]
patched = FragmentMetadata(
    id=f.id, files=f.files, physical_rows=f.physical_rows, deletion_file=f.deletion_file,
    row_id_meta=RowIdMeta.from_dict({"Inline": list(seq_range(target_rowid, target_rowid + 1))}),
)
op = lance.LanceOperation.Update(
    removed_fragment_ids=[], updated_fragments=[updated],
    new_fragments=[patched], fields_modified=[],
)

row holding id=2 has rowid 1
ids preserved: True
values correct: {'id': [1, 2, 3, 4], 'v': [10, 99, 30, 40]}

So we can unblock ourselves — thank you. On the delete-returns-row-ids half: we don't strictly need it, since we already scan the matched rows to run a UDF over them and can project _rowid in that same scan. It would be a convenience for callers deleting by predicate without a prior read.

Two questions, and one narrowed request.

1. Is the RowIdMeta.from_json/from_dict payload supported API, or internal? To get the above working I hand-encoded RowIdSequence { segments: [U64Segment { Range { start, end } }] } as varint protobuf. It round-trips byte-for-byte against a sequence Lance itself emitted:

lance's own sequence for a 3-row fragment holding ids 3,4,5: {'Inline': [10, 6, 10, 4, 8, 3, 16, 6]}
our encoder for Range(3, 6):                                 {'Inline': [10, 6, 10, 4, 8, 3, 16, 6]}

If that shape is internal and free to change, downstream consumers pinning a version floor would silently mis-assign row IDs on a future release — the exact invariant being protected. We're guarding it with a test that compares our encoder against a sequence Lance emits, so drift becomes a build failure rather than corruption. Knowing whether that guard is belt-and-braces or load-bearing would be useful.

2. For scattered IDs, is one Range segment per contiguous run the intended encoding, or is EncodedU64Array preferred past some density threshold? Our updates match arbitrary rows, so a sequence is generally many short runs rather than one range.

Narrowed request: given the mechanism already works, the remaining ask is just a public way to express it — RowIdMeta.from_ids([...]) or equivalent — so external writers don't hand-encode the protobuf. That's strictly smaller than the three options in the issue body, and it makes options (1) and (3) optional rather than necessary. Happy to send a PR for from_ids if you'd take it and can point me at where you'd want the validation (rejecting IDs not currently live, and duplicates).

[wjones127]

Here's the PR for a proper API to write row ids in Python: #8356