feat: route partial-schema merge_insert through the v2 write path

[wombatu-kun]

Summary

Closes #6442.

Partial-schema upserts (source column subset of the dataset schema) used to fall back to the legacy v1 Merger path, which wrote only the changed columns via Operation::Update with RewriteColumns. They now run through the same create_plan / FullSchemaMergeInsertExec pipeline as full-schema upserts. Missing source columns are filled from the target side of the join via a post-join projection, so the v2 writer sees a complete set of data columns and writes full rows as new fragments.

Per the issue, this intentionally does not replicate v1's RewriteColumns optimization — that is tracked separately in #4193. The v1 path is still used when the join key has a scalar index (falls through the old gate), so the legacy optimization remains available for the scalar-index-on-join-key case.

How it works

1. can_use_create_plan (gate) — accepts a source schema that is a type-compatible subset of the dataset schema. If insert_not_matched=InsertAll and any missing target column is non-nullable, the call short-circuits with an InvalidInput error naming the offending columns.
2. create_plan (plan construction) — after the join and the __action column, every dataset field missing from the source is added as an unqualified column populated from col("target.\"<name>\""). For matched rows this carries the existing target value; for unmatched source rows the outer join leaves it NULL.
3. necessary_children_exprs (projection pushdown) — the extension node now keeps unqualified columns whose name matches a dataset field, so the synthetic filled columns flow through to the write exec alongside source.*.
4. prepare_stream_schema — data columns are now emitted in dataset schema order keyed by name. This turns an accidental positional invariant into an explicit name-based one and is required for the partial-schema path (synthetic filled columns land at the end of the logical projection).

Consequences

• explain_plan() now works for partial-schema upserts.
• Bloom-filter conflict detection (inserted_rows_filter) is populated for partial-schema operations on unenforced primary keys; v1 always returned None.
• when_not_matched_by_source=Delete/DeleteIf is accepted for partial-schema sources (previously rejected outright with a NotSupported error).
• Partial-schema upserts with insert_not_matched=InsertAll now reject non-nullable missing columns at the API boundary with a descriptive error naming the offending columns, instead of producing a confusing downstream writer error. User-visible behavior change.

Tests

• Refactored test_merge_insert_subcols to branch on scalar_index: v1 structural assertions (tombstoned fields, preserved fragment ids) stay on the scalar_index=true branch; the scalar_index=false branch asserts v2 structural behavior. A shared key-lookup-based check verifies that columns not in source retain original values across both paths.
• Converted the old negative test_delete_not_supported into a positive test_delete_not_matched_by_source_on_v2_subcols test.
• Updated test_sub_schema_upsert_fragment_bitmap to reflect v2 semantics (3 fragments after upsert, both indexes preserved — v2 relies on unindexed-fragment fallback rather than eagerly invalidating the vector index).
• New test_merge_insert_subcols_v2_explain_plan confirms the v2 physical plan is used and that the filled other column appears in the projection.
• New test_merge_insert_subcols_v2_rejects_non_nullable_insert for the new validation error.
• New test_merge_insert_subcols_v2_bloom_filter for the bloom-filter conflict-detection acceptance criterion.
• Updated the Python test_merge_insert_subcols to drop v1-specific file-layout assertions while preserving the semantic data checks (column c unchanged for updated rows, NULL for newly inserted rows).

Test plan

• cargo check -p lance --lib --tests
• cargo clippy -p lance --lib --tests -- -D warnings
• cargo fmt --all
• cargo test -p lance --lib dataset::write::merge_insert — 127 passed
• cargo test -p lance --lib dataset::write — 190 passed
• cargo test -p lance --lib dataset:: — 1058 passed
• cargo test -p lance --lib index::tests — 63 passed
• Python tests (pytest python/python/tests/test_dataset.py::test_merge_insert_subcols) — run in CI

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 96.17647% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...lance/src/dataset/write/merge_insert/exec/write.rs	77.41%	6 Missing and 1 partial ⚠️
rust/lance/src/dataset/write/merge_insert.rs	98.05%	3 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

[wjones127]

Thanks for working on this. I'd like to see some tests to make sure this doesn't break in cases where there are camel case columns. Otherwise I think it looks good.

[wjones127]

issue(non-blocking): scanning these columns for all rows isn't ideal performance-wise. Consider the case where you are updating 3 rows with just part of the schema. This means you need to read all rows for missing columns in the table just to do this update. Ideally, we'd instead pull these columns in using TakeExec after the join.

I'm okay with keeping this code path for now, but if we do we should make a follow up ticket for optimizing to a Take later.

[wombatu-kun]

Agreed — materializing every missing column across the full target scan is wasteful when the update set is small. The follow-up is already tracked in #4193 ("Optimize upsert with partial schema and no index"), which is linked from this PR's description. I'll add a note on #4193 pointing at the post-join TakeExec approach you described so the optimization direction is captured concretely rather than as a generic "revisit later".

[wjones127]

This is looking good. Apologies for the delay in review. Thanks for working on this!

[wjones127]

Updated the branch. Will merge once CI is passing.