…tion (#6439) Source rows with NULL ON key columns were silently dropped because the action assignment logic used `ON_col IS NOT NULL` as a proxy for "source row is present in the join output". This conflates a legitimate NULL key with a NULL introduced by the outer join on the target side. Fix by injecting a `lit(true)` sentinel column into the source DataFrame before the join. After the join the sentinel is non-null for every source row and null only for target-only rows, making source row detection independent of ON column values. Strip the sentinel in `prepare_stream_schema` before writing and propagate it through projection pushdown in `necessary_children_exprs`. Before the join, inject a constant `lit(true)` column (`__merge_source_sentinel`) into every source row. After the join: - Source rows (whether matched or unmatched) → sentinel = true - Target-only rows (no source match) → sentinel = NULL (outer join NULL-fill) [assign_action](https://github.com/lance-format/lance/blob/6112a34bfe38618f07c099217dc3d89fd39ca6bb/rust/lance/src/dataset/write/merge_insert/assign_action.rs#L77) now uses sentinel IS NOT NULL to detect source row presence, making it correct regardless of what values the ON columns hold. The sentinel is a pure logical column — it never touches disk. It's stripped in [prepare_stream_schema](https://github.com/lance-format/lance/blob/6112a34bfe38618f07c099217dc3d89fd39ca6bb/rust/lance/src/dataset/write/merge_insert/exec/write.rs#L383) before any data is written, and [necessary_children_exprs](https://github.com/lance-format/lance/blob/6112a34bfe38618f07c099217dc3d89fd39ca6bb/rust/lance/src/dataset/write/merge_insert/logical_plan.rs#L148) is updated to propagate it through DataFusion's projection pushdown. Example that was broken before: ``` Target: (id=1, record_type="A") and (id=0, record_type=NULL) Source: (id=2, record_type=NULL) — new row, should be inserted ON: ["id", "record_type"] Old behavior: source row silently dropped (Action::Nothing) New behavior: source row correctly inserted (Action::Insert) ``` Fixes: #4644 --------- Signed-off-by: Pratik <pratikrocks.dey11@gmail.com> Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

merge_insert skips rows with NULL in on column #4644

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions