Stable-row-id Update/Merge: action for per-row version metadata refresh

[wjones127]

Parent PRD

Milestone: Action-based Transactions (UserOperation) — see milestone #11. Discussion: #5960. Design spike: #6448.

Background

#6454 (the final cutover) has a stated precondition: every operation has been migrated onto the action-level conflict resolver. That precondition is not met. update_to_actions and merge_to_add_fields (rust/lance/src/dataset/transaction/action.rs) return None — declining action translation — whenever manifest.uses_stable_row_ids() is true. actions_from_operation_with_manifest propagates that None, and ActionRebase::try_new surfaces it as Error::NotSupported.

Because commit_transaction routes every commit except a strict overwrite through the conflict resolver, cutting the production path over to ActionRebase and deleting TransactionRebase (as #6454 specifies) would make every Update/Merge commit on a stable-row-id dataset fail hard. Stable row IDs is a supported, increasingly-default feature — that is a severe regression and violates #6454's "no regression" acceptance criterion.

The root cause is a planning gap, not a design dead-end. #6843 ("representation of Update auxiliary fields") mapped the explicit fields of Operation::Update onto the action vocabulary. Per-row version metadata is not a field of Operation::Update — it is derived inside build_manifest from the stable-row-id counter and the current manifest version — so the field-by-field exercise never surfaced it, and #6448's §4 catalog was frozen at 19 actions without an action for it.

The gap

A stable-row-id Update/Merge refreshes per-row version metadata on fragments — created_at_version_meta and last_updated_at_version_meta — that no current Action reproduces. Three legacy code paths in build_manifest:

• resolve_update_version_metadata (transaction.rs) — sets created_at/last_updated on an Update's new fragments; created_at is traced back from the rows' source fragments.
• refresh_row_latest_update_meta_for_partial_frag_rewrite_cols — Update RewriteColumns, partially rewritten fragments.
• refresh_row_latest_update_meta_for_full_frag_rewrite_cols — Merge, physically rewritten / brand-new fragments.

The index-coverage re-pointing the update_to_actions comment also names is not part of this gap: RebindIndexCoverage (action A14) already exists and is already used for stable-row-id Rewrite. Stable-row-id Update RewriteRows can reuse it; the only nuance is reconciling A14's paired remove+insert with the legacy register_pure_rewrite_rows_update_frags_in_indices (insert-only, gated on the index covering all original fragments).

Why it is expressible

Action::apply takes &Manifest, so it is manifest-aware by design, and everything the refresh needs is manifest-derivable at apply time:

• The new version number is manifest.version + 1 — the trick UpdateMergedGenerations::apply already uses.
• created_at tracing reads the source fragments' version metadata; those fragments are still in the manifest as long as the refresh action is ordered before RemoveFragments (the action list is ordered — spike §3.3).
• The payload is effectively stateless ("refresh these fragments"), so rebase is a no-op and the action adds no conflict surface beyond fragments already claimed by AddFragments/RemoveFragments/UpdateDeletionVector.

What to build

• Add a new action — RefreshRowVersionMetadata — refreshing created_at_version_meta / last_updated_at_version_meta on a set of target fragments. Payload carries the target fragments and, for the partial-RewriteColumns case, the touched row offsets. Decide one parameterized action vs. separate full/partial actions.
• Implement its apply (porting all three legacy refresh paths), validate, reads, writes, and rebase (a no-op). Guarantee apply ordering before RemoveFragments so the created_at trace can read source fragments.
• The new action is in-memory only — no proto message. Actions here are transient (conflict resolution during a commit); the action-transaction wire format is not yet decided — PMC vote: new protobuf format for Action-based transactions #6455 is a pending PMC vote on it and Implement new protobuf serialization + feature flag #6456 implements serialization behind a feature flag, both after this issue and the Remove legacy Operation code paths #6454 cutover. The new action joins serialization in Implement new protobuf serialization + feature flag #6456.
• Make update_to_actions and merge_to_add_fields stop bailing on uses_stable_row_ids(): emit the new action, and for Update RewriteRows additionally emit RebindIndexCoverage. Reconcile the A14 / legacy index-rebind semantics.
• Extend the differential matrix with stable-row-id Update/Merge generators (currently excluded — the matrix is "N×N" only over translatable shapes). Confirm the new resolver agrees with the legacy oracle.
• Update the design doc: unfreeze §4 to 20 actions with the new catalog row (payload, read-set, write-set); rewrite the §5 "stable-row-id bail" paragraphs for Update and Merge; resolve the §5/§11 contradiction (§5 documents a permanent legacy fallback, §11 says legacy is deleted at cutover); record the rebase-correctness argument.

conflict_resolver.rs must not be modified — it remains the production path and the differential oracle until the #6454 cutover.

Out of scope: the file-level deletion-vector auto-merge divergence (design doc §5 divergence (1)) — orthogonal, non-fatal (retryable, eventually correct), and remains deferred.

Acceptance criteria

• New RefreshRowVersionMetadata action defined; apply reproduces all three legacy refresh paths
• apply/validate/reads/writes/rebase implemented; rebase-correctness argument recorded in the design doc
• update_to_actions / merge_to_add_fields no longer bail on uses_stable_row_ids()
• RebindIndexCoverage / legacy index-rebind semantics reconciled for stable-row-id Update RewriteRows
• ActionRebase::try_new no longer returns Error::NotSupported for any stable-row-id Update/Merge
• Differential matrix extended with stable-row-id Update/Merge generators; green
• Design doc §4/§5/§11 updated; §5/§11 contradiction resolved
• Existing stable-row-id update / merge tests still pass
• conflict_resolver.rs unmodified
• cargo fmt --all; cargo clippy --all --tests --benches -- -D warnings clean

Blocked by

• Update: action translation + conflict resolution #6898 (Update — action translation + conflict resolution)
• DataReplacement: ReplaceFragmentColumns action + translation + conflict resolution #6842 (DataReplacement — ReplaceFragmentColumns + conflict resolution)

Blocks

• Remove legacy Operation code paths #6454 (final cutover) — its "every operation migrated" precondition holds only once this lands.

User stories addressed

• User story 6: single code path for conflict resolution
• User story 9: old operations translated to actions