RFC: BatchCommitTables for DirectoryNamespace - Multi-Dataset Atomic Metadata Commits

[aaltshuler]

RFC: BatchCommitTables for DirectoryNamespace — Multi-Dataset Atomic Metadata Commits

Responds to: #6668 "Multi-dataset atomic commit primitive" (@ragnorc, 2026-05-02)
Status: Request for Comments
Type: Namespace / Storage Engine
Disclosure: @ragnorc and I are building Omnigraph together; nanograph (also cited as an affected consumer in §1) is our project as well. Flagging up front so the consumer list is read with that context.
Verified against: lance v6.0.0 stable, lance-namespace v0.7.6

---

TL;DR

The Lance namespace spec defines BatchCommitTables for atomic multi-table metadata commits. The OpenAPI pipeline ships generated wire-format clients in Rust, Python, and Java. No server-side implementation exists in any language. The Rust LanceNamespace trait does not declare the method; the Java core throws UnsupportedOperationException.

#6668 asks for exactly this. This RFC proposes how to land it in DirectoryNamespace using primitives that already exist (__manifest Lance dataset + single-dataset OCC + copy_if_not_exists). Of the three additional requirements raised in #6668: Phase 1 delivers namespace-view atomicity for namespace-managed readers (covering the failure-semantics ask, §5.3); Phase 2 adds branch-awareness (§5.1); Phase 3 closes the per-table HEAD visibility leak for direct Lance readers (§5.2), which depends on a new two-phase Dataset publish primitive — that's Lance-core work, not just namespace wiring.

The contribution is at the namespace layer (cross-table). It is orthogonal to milestone #11 (table-layer composable transactions, UserOperation/Action) — sibling primitives, not stacked. Either can ship independently; both work better together; neither depends on the other for correctness.

---

1. Motivation

[Restating from #6668 for context.] CommitBuilder::execute(transaction) is per-dataset. There is no public Rust primitive for atomic multi-dataset metadata commits, even though the namespace specification and the generated REST client define one. Consumers maintaining invariants across multiple Lance datasets currently use a coordinator-dataset pattern, which leaves a partial-success window between per-dataset commits and the coordinator commit.

Consumers affected (from #6668, annotated):

• Graph databases that map node and edge types to one Lance dataset each. (Omnigraph; nanograph.)
• Snapshot / branching systems that need every dataset's commit to land before exposing a new published view.
• Pipelines that update a base dataset and a derived index dataset in lockstep.
• Multi-tenant systems that need bulk migrations to apply across N tenants atomically.

The Omnigraph case concretely: a typed property graph where each node/edge type lives in its own Lance dataset. A graph mutation touching N tables currently runs:

1. commit_staged per table — N sequential Lance dataset advances, each individually visible to readers
2. __manifest row-level CAS to publish the multi-table state via ManifestBatchPublisher::publish

Step (1) leaks per-table HEAD visibility before step (2) lands. The "Lance HEAD ahead of __manifest" drift class — for which we ship an open-time recovery sweep — would not exist if the substrate owned multi-table atomicity. The recovery sweep is consumer-side workaround for the absence of this primitive.

2. Where the substrate already is

The atomic primitive BatchCommitTables needs is in place:

• __manifest Lance dataset. DirectoryNamespace maintains this at {namespace_root}/__manifest/. Table-version visibility is derived from table_version rows in __manifest. Single-dataset OCC arbitrates concurrent writes via copy_if_not_exists on the _versions/<n>.manifest file. Any change that updates K rows in __manifest in one dataset commit is atomic across those K rows.

• Generated wire types. BatchCommitTablesRequest, CommitTableOperation, and the four sub-requests (DeclareTableRequest, CreateTableVersionRequest, BatchDeleteTableVersionsRequest, DeregisterTableRequest) have shipped as auto-generated Rust/Python/Java models since lance-namespace v0.6.0. Shape unchanged through v0.7.6.

• MergeInsertBuilder with WhenMatched::Fail on object_id — already used by insert_into_manifest_with_metadata for multi-row inserts into __manifest.

• copy_if_not_exists across object stores (local FS, S3, GCS, Azure Blob via the ObjectStore abstraction) — the same primitive create_table_version already uses.

For namespace-managed readers, the gap is wiring, not new mechanisms. Full closure of the per-table HEAD visibility window for direct Lance readers needs additional Lance-core work — see §5.2.

3. What's missing

• Trait method. rust/lance-namespace/src/namespace.rs declares 40+ methods; batch_commit_tables is not one of them. Only batch_delete_table_versions (single-table) is present.
• DirectoryNamespace impl composing the four operation variants into a single __manifest mutation.
• RestNamespace adapter routing the call to the existing REST endpoint surface.
• Cross-language bindings — Python core, Java core, JNI.

Non-trivial constraint: BatchCommitTables must not be implemented as N parallel create_table_version calls. The existing single-table path records the __manifest row best-effort — it logs a warning if the row insertion fails. For BatchCommitTables the row mutation must be mandatory and atomic, or the "__manifest as source of truth" claim in §5.3 doesn't hold. The implementation needs to share create_table_version's lower-level physical-copy helpers but route through a different commit boundary.

4. Proposed design

4.1 Trait method

async fn batch_commit_tables(
    &self,
    _request: BatchCommitTablesRequest,
) -> Result<BatchCommitTablesResponse> {
    Err(Error::not_supported("batch_commit_tables not implemented"))
}

Default impl returns not_supported, matching the convention for other optional ops (alter_transaction, update_table).

4.2 DirectoryNamespace::batch_commit_tables

Invariant: no file appears at a canonical _versions/<N>.manifest path until its corresponding __manifest row commits. Per-table version manifests are written to a per-batch staging directory first, then promoted via rename_if_not_exists only after __manifest row insertion succeeds.

async fn batch_commit_tables(
    &self,
    request: BatchCommitTablesRequest,
) -> Result<BatchCommitTablesResponse> {
    self.require_table_version_storage_enabled()?;
    self.validate_operation_set(&request.operations)?;

    let batch_uuid = Uuid::new_v4();
    let mut retry_count = 0;
    let mut staged_versions: Vec<StagedVersionFile> = Vec::new();

    loop {
        let manifest_snapshot = self.read_manifest_dataset_snapshot().await?;
        let plan = self.plan_manifest_batch(&request.operations, &manifest_snapshot)?;

        // Stage: write per-table version manifests to _staging/<batch_uuid>/,
        // NOT to canonical _versions/ paths. Direct Lance readers cannot
        // observe HEAD advance because the canonical path is still empty.
        staged_versions = self.stage_version_manifests(&plan.create_versions, batch_uuid).await?;

        // Atomic publish: one __manifest dataset commit composing all
        // requested row-level changes (insert table_version rows for
        // CreateTableVersion, insert table rows for DeclareTable, delete
        // for DeleteTableVersions/DeregisterTable). Conflicts here are
        // wrapped as ConcurrentModification at the namespace surface.
        match self.commit_manifest_mutation(plan.manifest_mutation, manifest_snapshot.version()).await {
            Ok(commit_result) => {
                // Promote: rename _staging/<batch_uuid>/ files to canonical
                // _versions/<N>.manifest paths. At this point the __manifest
                // row is durable; promotion failure leaves the row valid
                // and the canonical file appears on next operation.
                self.promote_staged_versions(&staged_versions).await?;
                return Ok(self.build_response(commit_result, &request.operations));
            }
            Err(Error::ConcurrentModification { .. }) if retry_count < self.commit_retries => {
                // __manifest advanced; staging files harmless (canonical
                // paths never touched). Best-effort cleanup; retry.
                self.cleanup_staged_versions(&staged_versions).await.ok();
                staged_versions.clear();
                retry_count += 1;
                continue;
            }
            Err(e) => {
                self.cleanup_staged_versions(&staged_versions).await.ok();
                return Err(e);
            }
        }
    }
}

Three atomic boundaries, ordered:

1. Stage — per-table version manifests land in _staging/<batch_uuid>/. Failure here is benign; nothing visible.
2. Commit __manifest — single Lance dataset commit inserts/updates/deletes the table_version rows. This is the atomicity point: either all row changes flip visible together, or none do. Mixed insert + delete in one batch is supported by Lance's existing CommitBuilder::execute_batch(Vec<Transaction>) (per #3734) — multiple per-dataset transactions are atomically published in one manifest version bump, so a BatchCommitTables request with CreateTableVersion + DeregisterTable ops needs no new __manifest mutator.
3. Promote — staged files rename to canonical _versions/ paths via rename_if_not_exists. After step 2 succeeds, the __manifest row is the durable source of truth; promotion is recoverable on the next operation if it fails mid-step.

A reader through the namespace sees nothing until step 2 commits. A reader bypassing the namespace and opening per-table datasets directly sees per-table HEADs advance during step 3 — one rename at a time, in whatever order the runtime promotes them. This means:

• Failed batches leave no canonical _versions/<N>.manifest files at all (step 1 wrote only to _staging/). The §5.3 "no observable state change to readers" claim holds at this boundary.
• Successful batches still leak per-table HEAD ordering to direct Lance readers during step 3. The leak window shrinks from "duration of staging + manifest commit" (current Omnigraph) to "duration of N renames" (much smaller), but is not zero.
• Full closure for direct Lance readers requires the two-phase Dataset primitive discussed in §5.2 — Phase 3 of this RFC.

4.3 Conflict model: snapshot isolation per batch

Each batch reads __manifest at a single dataset version, validates all operations against that snapshot, and commits as a single dataset advance. Concurrent batches race for the same advance; a loser retries from a fresh snapshot. This matches Lance's existing single-dataset OCC, applied to __manifest.

Properties:

• Operations validated against a consistent snapshot (no torn reads).
• The batch is serializable with respect to other batches and other single-table operations on the same __manifest.
• A failed batch produces no observable state change to readers (orphan version files exist but are not referenced by __manifest).
• Retries preserve correctness — each retry reads a fresh snapshot.

4.4 Cross-backend portability

The implementation depends on three primitives Lance already provides:

1. Dataset::commit on __manifest — works on all supported backends today.
2. copy_if_not_exists / rename_if_not_exists for staging and promotion — works on all four major object stores via the ObjectStore abstraction (with the existing fallback path for backends that lack native conditional-write support; the BatchCommitTables impl inherits whatever fallback shape Lance already uses for create_table_version).
3. Filesystem-style listing for orphan cleanup.

No new backend-specific code. Behavior on each backend is determined by the existing ObjectStore trait impls.

5. Addressing #6668's three additional requirements

#6668 raises three requirements beyond "implement what the spec says."

5.1 Branch-awareness

Non-main branch publication without flattening branch state. Phase 1 ships main-only; Phase 2 adds a first-class branch parameter on BatchCommitTablesRequest, contributed as a spec PR alongside the implementation. __manifest CAS becomes scoped per (table, branch). Lance has native branch primitives at the dataset layer; for BatchCommitTables to interact with them portably across namespace implementations, branch must be in the spec — not in context, which is documented as implementation-custom. A lance-branch: context-key encoding is a transitional stopgap if the spec PR lags the implementation, but is not the long-term design.

5.2 Publish must accept staged manifests without per-table HEAD visibility

The most architecturally significant of the three. Per-table commit_staged must not advance Lance HEAD until the multi-table __manifest commit succeeds. Today's Dataset::commit does not separate stage from publish, so closing the gap requires Lance-core work — either alongside #6658 (two-phase delete) and #6666 (two-phase vector index), or via a new Dataset::stage_commit primitive.

Without one, Phase 1 leaks per-table HEAD between staging and the __manifest flip. Reads via the namespace see consistent multi-table state; reads that bypass the namespace and open per-table datasets directly see HEADs advance one-by-one. This is what Omnigraph runs today.

5.3 Explicit failure semantics; __manifest as source of truth

A failed batch leaves orphan per-table version files invisible to namespace-managed readers (no table_version row points at them). Cleanup tiers:

• Phase 1 (eager): next operation against the affected table scans for orphans and deletes unreferenced ones.
• Phase 2 (lazy): background reaper at configurable interval.
• Phase 3 (reference-based, ideal): version files carry a batch identifier; reaper deletes files whose batch did not commit. Deterministic; no scanning.

At every phase, __manifest is authoritative — files on disk not referenced in it don't exist for namespace reads.

6. Relationship to milestone #11 (UserOperation/Action)

BatchCommitTables is at the namespace layer (cross-table); milestone #11 (#5960's UserOperation { repeated Action }) is at the table layer (intra-table). They are sibling primitives at different architectural layers, not stacked.

• Milestone Compute min/max values for columns to support file-level and chunk-level pruning. #11 lets a single Lance dataset commit atomically combine multiple actions (e.g., AddFragments + AddIndex on one table). It does not address atomicity across multiple datasets.
• BatchCommitTables lets the namespace-managed view of multiple tables advance atomically — table_version rows in __manifest publish together via row-level CAS on a single Lance dataset. The per-table underlying Lance HEADs still advance one-by-one during the staging phase; closing that visibility window requires the two-phase Dataset publish primitive discussed in §5.2. It does not address intra-table composability.

A consumer might want both: "atomically (a) append data + create index on table A, AND (b) append data on table B." BatchCommitTables provides the cross-table namespace-view grouping (b); milestone #11 provides the intra-table composability (a). Both can ship independently; both work better together.

This proposal does not block on milestone #11 and does not propose any change to it. The two land on independent timelines.

7. Phasing

Phase	Scope	Deliverable
1. Trait + minimal impl + bindings	Trait methods for both batch_commit_tables and batch_create_table_versions (matching the full #6668 ask) + DirectoryNamespace impl for CreateTableVersion and DeclareTable (main branch only; eager cleanup) + RestNamespace adapter + Python core + Java core + JNI bindings, matching the #6678 "backfill and refresh" cross-language PR shape	Both batch ops on the trait; multi-table version advance works end-to-end across languages; __manifest row-level CAS atomic across N tables
2. Full operation coverage + branches	Add DeleteTableVersions, DeregisterTable; add branch-awareness per §5.1	All four spec ops; non-main branches
3. Two-phase publish	Resolve §5.2 — either depend on a new Dataset::stage_commit primitive or coordinate per-table staging through __manifest	No per-table HEAD visibility leak before namespace metadata is durable
4. Reference-based GC	Replace eager cleanup with batch-id-tagged version files + reaper	Deterministic crash recovery without per-table scanning

Phase 1 ships cross-language bindings together with the Rust impl (per the #6678 template); Phases 2–3 are the substantive remainder responding to #6668. Phase 4 is polish.

8. Open questions

1. Branch encoding (§5.1). The proposal commits to a first-class branch parameter via spec PR (not context-key). Open question: is the spec PR straightforward to add to lance-namespace, or are there structural blockers that would push us toward keeping the transitional context-key encoding longer?

2. Two-phase publish on Dataset (§5.2). Does this work fit alongside Expose two-phase delete API (DeleteJob::execute_uncommitted analog) #6658 (two-phase delete) and Expose build_index_metadata_from_segments (or commit_existing_index_segments) for two-phase vector-index commits outside the lance crate #6666 (two-phase vector index), or does it want a new Dataset::stage_commit primitive? Without one, Phase 3 cannot close the per-table HEAD visibility window.

3. Surface alternative — MultiCommitBuilder on Dataset. Multi-dataset atomic commit primitive #6668 explicitly leaves the API shape open: "If a different surface is preferred — e.g., a MultiCommitBuilder on Dataset rather than methods on LanceNamespace — happy to discuss. The architectural ask is 'make the substrate own multi-table atomicity' rather than the specific API shape." This proposal pursues the trait-method shape because it matches the already-shipped namespace spec, the REST surface, and the auto-generated Rust/Python/Java client code through v0.7.6 — wiring an existing surface, not designing a new one. A Dataset-level MultiCommitBuilder would either require an additional spec at the Dataset layer or a routing shim from the namespace API to a Dataset primitive — strictly more spec work than the trait shape. Open question: do maintainers see structural problems with the trait shape that would justify the additional Dataset-level work?

Settled in this RFC (no longer open):

• Mixed insert + delete in one __manifest commit. Resolved in §4.2 — CommitBuilder::execute_batch(Vec<Transaction>) already supports this; no new mutator needed in DirectoryNamespace.
• expected_current_version on CreateTableVersionRequest. Omnigraph runs without it via application-layer expected_table_versions CAS; the gap isn't blocking for Multi-dataset atomic commit primitive #6668. A spec extension would be cleaner but is a separable follow-up RFC; not blocking this contribution.
• BatchCreateTableVersions and BatchCommitTables coexistence. Both ship on the trait in Phase 1 per Multi-dataset atomic commit primitive #6668's ask. Deprecation of the older type can be a follow-up RFC once BatchCommitTables is established as the canonical successor.

9. References

• Upstream ask this responds to: #6668 Multi-dataset atomic commit primitive (@ragnorc, 2026-05-02)
• Process: #5517 What is the process for making a specification change?; lance.org/community/voting
• Spec: Lance Namespace Operations; Directory namespace impl-spec; BatchCreateTableVersions
• Spec PR that introduced BatchCommitTables: lance-namespace#315 (in v0.6.0)
• Generated client (v0.7.6): batch_commit_tables_request.rs; commit_table_operation.rs
• Substrate: DirectoryNamespace::create_table_version at dir.rs (v6.0.0)
• Related namespace roadmap: lance-namespace#180 (multi-table multi-statement transaction); lance-namespace#168 (tracker)
• Related blocker issues: #6658 (two-phase delete API); #6666 (two-phase vector index)
• Namespace area structure: #6109 Clearly separate Lance Namespace and Lance Catalog
• Adjacent table-layer work (sibling, not stacked): #3734; #5960 + milestone #11; #5952

10. Request for Comments

The three feedback points that most shape what this contribution looks like:

1. Two-phase publish (§5.2 / §8 Q2). Does the "no per-table HEAD visibility leak" requirement from Multi-dataset atomic commit primitive #6668 fit alongside Expose two-phase delete API (DeleteJob::execute_uncommitted analog) #6658 (two-phase delete) and Expose build_index_metadata_from_segments (or commit_existing_index_segments) for two-phase vector-index commits outside the lance crate #6666 (two-phase vector index) as the same primitive, or does it want a separate Dataset::stage_commit? Without one, Phase 3 cannot close the visibility window for direct Lance readers.
2. Surface alternative (§8 Q3). Trait method on LanceNamespace (current proposal, matches existing namespace spec + REST surface), or MultiCommitBuilder on Dataset (the alternative Multi-dataset atomic commit primitive #6668 raises)? The architectural ask is the same; the API shape is open.
3. Layer framing (§6). Does the namespace-vs-table layer distinction (this proposal sibling-to, not stacked-on, milestone Compute min/max values for columns to support file-level and chunk-level pruning. #11) match how the team thinks about this? Getting this right early avoids design churn later.

The remaining §8 open question — branch encoding (Q1) — is a scoped spec-PR feasibility check; happy to take maintainer guidance during the spec/impl PRs rather than blocking design alignment here.

I'm interested in contributing toward the implementation and happy to coordinate with @ragnorc on scope and ownership; what makes sense from the maintainer side is also welcome input.

[aaltshuler]

@jackye1995
Following up from the may 21th meetup: here's what I found going through recent WAL / commit-path activity to see how it lands against this RFC.

The way I understand it now: MemWAL is intra-table (LSM tree, memtable → L0 flushed generations, streaming writes into one Lance dataset) and this RFC is cross-table namespace-layer atomicity, so the two don't directly overlap. But the broader substrate around Phase 3 has moved in ways that matter — two precedents that didn't exist when I posted, and one blocker that's no longer a blocker.

On Phase 3 substrate (§5.2 / §8 Q2): #6658 is closed, landed as #6781 (feat: expose uncommitted delete transactions, merged 2026-05-14). #6666 is still open but #6806 (feat: expose staged index segment transactions) is carrying the same staged-transaction shape across. §8 Q2 sharpens — it's no longer "should #6658 and #6666 be the same primitive as multi-table publish?" but "the staged-transaction shape from #6781 is what #6666 is about to mirror, so does multi-table want to plug into that, or want its own surface?"

Concretely the gap is that per-table commit_staged lands the new manifest in _versions/ before the __manifest row commits — so what Phase 3 needs is a way to stage a per-table manifest without publishing it to _versions/ until the cross-table commit lands. That's the same "produce an uncommitted transaction, commit it elsewhere" shape as #6781 and #6806, which is why I think they're the right primitive to generalize rather than a separate stage_commit.

Two precedents I'd fold into a revision:

• fix(mem_wal): advance visibility cursor on WAL durability, not indexing #6754 (fix(mem_wal): advance visibility cursor on WAL durability, not indexing, merged) explicitly adopts durability-before-visibility intra-table — the visibility cursor advances when the WAL is durable, not when indexing finishes. §4.2's staging → promote is the namespace-layer cousin of exactly that invariant. Reframing §4.2 in that vocabulary would help reviewers see the proposal as "extend an existing Lance pattern" rather than "novel idea."

• feat: switch commit strategy to optimistic by default #6836 is moving to make optimistic-first commit the default on the Dataset commit loop (the title and approach changed last week — it dropped the CommitStrategy enum framing in favor of simply skipping rebase until the first conflict, needs_rebase = backoff.attempt() > 0). No configurable knob, but the concept aligns directly with the snapshot-isolated retry loop in §4.3.

§5.2 feels like it's narrowing rather than stalled, so I'm ready to start sketching Phase 1.

@jackye1995 two questions where your read would help:
(1) sequencing — does Phase 1 wait for #6806 to land, or is it fine to start against the current substrate?
(2) the §8 Q3 surface choice — trait method on LanceNamespace (current proposal) or MultiCommitBuilder on Dataset (the alternative #6668 raised)? For me either direction works to draft against. (Tangential signal: lancedb's new per-table branch API in lancedb#3490 went with builder methods on Table/Connection rather than namespace-trait methods — though that's a read/checkout surface, so I'm not sure it transfers to a publish primitive.)