feat(mem_wal): support delete against non-nullable base columns

[hamersaw]

Problem

A tombstone carries the primary key and null in every other column, so ShardWriter::delete required every non-PK column to be nullable in the base table. That pushes a storage-engine detail into the user's schema for no reason the user can see.

Approach

Split the shard's schema in two:

• logical — the base table's schema, exactly as the caller declared it. The contract input is validated against, and the schema the scan path returns.
• storage — every non-PK top-level field widened to nullable. What the memtable, WAL entries, and SSTables physically carry.

This mirrors the logical/physical split SchemaAdapter already applies to JSON and view types in dataset/utils.rs, and rides the boundary that already exists — _tombstone is a physical column the SSTable schema carries and the base table does not.

Widening is top-level only. Arrow validates nullability just at the top level of a RecordBatch, so a null FixedSizeList/Struct needs no change below the top; a vector column's item field is untouched and gains no validity layer. Primary keys are never widened (Schema::unenforced_primary_key requires them non-nullable), so build_tombstone_batch still rejects a null, mistyped, or missing key — the delete path needed no new validation.

Where the contract is enforced

Ingress — put validates against the logical schema before the WAL append.

This is now the only gate. Both append (write/insert.rs) and merge_insert compare schemas with NullabilityComparison::Ignore, and the encoder derives validity from the array rather than the field, so a null that got past put would land in a non-nullable base column silently. Validating pre-append also matters for a second reason: a batch that is appended and only then rejected fails identically on every replay, leaving the shard unable to reopen — the same hazard that puts validate_index_configs ahead of claim_epoch.

WAL-only mode is covered as well; it previously validated nothing at all.

Egress — the scan narrows back to the logical schema after tombstones are filtered.

project_to_canonical documented that it emits its target_schema but did not: DataFusion derives ProjectionExec nullability from its expressions, not from the requested schema. A new SchemaRelabelExec makes that real. The same node widens arms so they agree before UnionExec/CoalesceFirstExec, both of which require exact schema equality — CoalesceFirstExec::new asserts it and would otherwise panic on the base-arm/WAL-arm nullability difference.

The narrowing doubles as a runtime assertion: if a tombstone ever escaped its filter, RecordBatch::try_new rejects the null instead of handing the caller a row of nulls.

Ordering falls out of this — carry_schema in the point-lookup path is built on the widened schema, because tombstones are still in flight until filter_tombstones_after_coalesce. vector_search and fts_search needed no changes; they already route every arm through project_to_canonical.

ensure_tombstone_column now always re-labels instead of passing through a batch that already has the column, so an entry written under an older storage schema replays into the current one.

Tests

13 new tests; 575 mem_wal lib tests pass.

• test_delete_against_non_nullable_base_column_round_trip — the headline: delete against a base table with a non-nullable non-PK column, survivors keep their values, and the scan reports the base table's own nullability.
• test_put_rejects_null_in_non_nullable_base_column / ..._wal_only_... — the ingress gate in both modes.
• test_build_tombstone_batch_nulls_non_nullable_base_column / ..._rejects_null_primary_key — widening works, PKs still strict.
• relax_* — top-level-only widening, nested fields untouched, _tombstone stays non-nullable, idempotence, metadata preserved (the PK marker rides on field metadata).
• schema_relabel::tests — widening, narrowing, narrowing rejects a surviving null, empty batches.

Not yet run

Draft because these are outstanding, not because the change is incomplete:

• Full cargo test -p lance --lib (~2700 tests) — only the mem_wal subset has been run.
• cargo clippy --all --tests --benches -- -D warnings.
• No end-to-end test yet confirming that Lance silently accepts a null into a non-nullable base column via merge_insert. The ingress gate is written as if it is load-bearing for base-table integrity, which is the safe assumption and what the code reading indicates, but it is unverified.

cargo fmt --all has been run.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[lance-gatekeeper]

✅ Gate recommendation: approve.

The change fixes the mismatch between the logical schema contract and tombstone storage at the right boundary. It keeps caller validation and scan output strict while widening only the physical non-key fields needed by delete markers, and it normalizes replay and mixed-source reads without changing a stable format.