Lance Field Assignment

[Xuanwo]

Summary

Lance should optionally track FieldAssignment: for a field in a dataset snapshot, whether each live row has a current logical assignment. Assignment is independent of Arrow validity and physical storage, so an explicitly assigned NULL is distinct from an unassigned placeholder or stale stored value.

Assignment is exposed through is_assigned(field), a first-class, non-null boolean Lance expression. Logical field writes create assignments and explicit invalidation clears them. Lance owns this snapshot state across mutations and physical rewrites; callers own computations, dependencies, freshness, correctness, and scheduling.

Problem

A nullable field cannot distinguish an omitted or invalidated value from an explicitly assigned NULL. A separate boolean field plus bitmap index can encode that distinction, but exposes storage state as user data and makes every caller maintain sparse updates, index consistency, snapshots, and compaction remapping.

Lance needs one exact assignment bit per live row, strongly bound to a stable field ID. The bit carries no failure state, producer identity, dependency graph, provenance, or workflow metadata.

Proposed Design

Public Semantics

Assignment	Arrow validity	Meaning
Unassigned	Either	The field has no current logical assignment for this row
Assigned	Invalid	The field was explicitly assigned NULL
Assigned	Valid	The field was explicitly assigned a non-NULL value

Assignment is snapshot state, not a record of whether the field was ever written. A successful logical write assigns the field for the affected row; a later explicit invalidation makes it unassigned. Renaming preserves the state because its descriptor is bound to the stable field ID. Assignment does not claim that a value is fresh, correct, or produced by a particular computation.

Assignment tracking is opt-in. Creating it requires an explicit initial state of unassigned or assigned; Lance never infers assignment from NULLs or file layout. Initializing an existing field as assigned is a migration assertion by the caller. Ordinary projection continues to return stored Arrow values regardless of assignment state.

The proposed Python schema extensions are:

dataset.add_columns(
    pa.field("embedding", pa.list_(pa.float32(), 1536)),
    assignment="unassigned",
)

dataset.alter_columns({
    "path": "thumbnail",
    "assignment": "assigned",
})

Adding assignment tracking to an already tracked field is an error. After initialization, state changes occur through field writes and invalidations rather than repeated schema alteration.

Expression Contract

is_assigned(field) is a regular boolean expression in Lance's expression model:

scanner = dataset.scanner(
    columns={
        "id": "id",
        "embedding": "embedding",
        "embedding_assigned": "is_assigned(embedding)",
    },
    filter="is_assigned(embedding) OR priority > 10",
)

The function accepts exactly one field reference. Planning resolves that reference to a stable field ID in the input snapshot. The result is always true or false; an unknown field, an untracked field, or a non-field argument is a planning error.

is_assigned has the same compositional contract as other boolean expressions. It may appear in projection, filtering, aggregation, ordering, NOT, AND, OR, and nested expressions. Query meaning never depends on whether the optimizer recognizes a pushdown opportunity.

Lance represents the function as a native logical expression in DataFusion and as a Lance extension function in Substrait. Python and other bindings expose the same expression through their Lance expression builders. A standalone Arrow array cannot evaluate assignment because it has no dataset snapshot or row identity; the Lance scanner binds the expression and may materialize an ordinary BooleanArray for downstream Arrow evaluation.

Planning and Execution

When is_assigned participates in a filter, the planner may evaluate it as an exact row mask and combine it with deletion masks, scalar-index results, ZoneMap pruning, or vector-search prefilters. For example, is_assigned(embedding) AND language = 'en' can prune unassigned rows before reading language or embedding.

Pushdown is an optimization, not the only execution path. For projection or an expression such as is_assigned(embedding) OR priority > 10, the scan operator can produce the required BooleanArray from snapshot assignment state and each batch's physical row positions. This fallback preserves arbitrary expression semantics without reading the assigned field's values. A query that does not reference is_assigned performs no assignment-state I/O.

Mutation Contract

A successful logical write of a tracked field assigns only the rows for which that field was supplied, including supplied NULLs. Append or insert leaves a row unassigned when it omits the field. Physical rewrites, including compaction, never create assignments merely because they rewrite value bytes.

External batch computation uses the existing merge-insert workflow. This proposal extends update-only merge_insert to accept _rowid, matching existing merge support:

pending = dataset.scanner(
    columns=["_rowid", "text"],
    filter="NOT is_assigned(embedding)",
)

for batch in pending.to_batches():
    result = pa.table({
        "_rowid": batch["_rowid"],
        "embedding": embed(batch["text"]),
    })
    (
        dataset.merge_insert(on="_rowid")
        .when_matched_update_all()
        .execute(result)
    )

Only matched rows are written and assigned. Mutation statistics expose rows that no longer match.

Existing mutation APIs gain invalidate_fields, which clears assignment for the rows changed by the mutation in the same commit:

dataset.update(
    updates={"text": "'replacement text'"},
    where="id = 42",
    invalidate_fields=["embedding"],
)

An invalidation-only mutation may omit value expressions. A mutation cannot both write and invalidate the same field. After initialization, callers cannot assign state directly; only a successful field write creates an assignment. Lance does not infer dependencies or reject stale computations. If changing text invalidates embedding, the caller must request that transition and enforce any source-read validation it needs.

Snapshot Storage

The manifest contains an optional FieldAssignmentState section keyed by stable field ID. Each field points to an immutable root with compact per-fragment states for no rows assigned, all rows assigned, or a partial set represented by compressed physical-row-offset bitmaps. New snapshots reuse unchanged roots and bitmap pages and replace only touched fragment entries.

Deletion applies the live-row mask without rewriting assignment state. A rewrite preserves membership when row positions survive; compaction or relocation remaps it with the same old-to-new row mapping used for data. Assignment is authoritative snapshot state rather than a scalar index, so it has no asynchronous build, catch-up, or rebuild lifecycle. Its execution may reuse RowAddrMask, IndexExprResult, and ZoneMap-like fragment pruning.

Data Overlay's existing FieldCoverage remains mutation-local: it records which row-field cells an overlay file supplies. It neither represents nor replaces snapshot-level FieldAssignmentState.

Cost and Compatibility

Ordinary scans and takes incur no assignment cost. An assignment expression reads the referenced fields' roots and only the partial-fragment bitmaps it needs. Field writes and invalidations update touched fragment entries; time travel reuses immutable snapshot state. Compaction is the principal scaling risk because a row-address-changing rewrite must remap every partial state for the affected tracked fields.

Before stabilization, benchmarks must compare this design with a boolean field plus bitmap index on metadata size, expression and planner latency, sparse-update amplification, remote I/O, projection throughput, and compaction cost.

Assignment descriptors are required-writer feature markers. Older readers may perform ordinary Arrow projections, but cannot evaluate is_assigned; writers that cannot preserve assignments must fail before mutation. DataFusion and Substrait consumers must advertise support for the extension expression. Before creation is enabled, append, update, merge, merge-insert, delete, schema evolution, restore, and compaction paths must all preserve the contract.

[Xuanwo]

Created #8531 as a PoC