fix(merge_insert): use sentinel column for NULL-safe source row detection

[pratik0316]

Source rows with NULL ON key columns were silently dropped because the action assignment logic used ON_col IS NOT NULL as a proxy for "source row is present in the join output". This conflates a legitimate NULL key with a NULL introduced by the outer join on the target side.

Fix by injecting a lit(true) sentinel column into the source DataFrame before the join. After the join the sentinel is non-null for every source row and null only for target-only rows, making source row detection independent of ON column values.

Strip the sentinel in prepare_stream_schema before writing and propagate it through projection pushdown in necessary_children_exprs.

Before the join, inject a constant lit(true) column (__merge_source_sentinel) into every source row. After the join:

• Source rows (whether matched or unmatched) → sentinel = true
• Target-only rows (no source match) → sentinel = NULL (outer join NULL-fill)
  assign_action now uses sentinel IS NOT NULL to detect source row presence, making it correct regardless of what values the ON columns hold.

The sentinel is a pure logical column — it never touches disk. It's stripped in prepare_stream_schema before any data is written, and necessary_children_exprs is updated to propagate it through DataFusion's projection pushdown.

Example that was broken before:

Target: (id=1, record_type="A") and (id=0, record_type=NULL)
Source: (id=2, record_type=NULL) — new row, should be inserted
ON: ["id", "record_type"]
Old behavior: source row silently dropped (Action::Nothing)
New behavior: source row correctly inserted (Action::Insert)

Fixes: #4644

[pratik0316]

Hi @wjones127 can you pls provide a review

[wjones127]

This looks like a nice fix. There's some optional simplifications you can do with the tests. I've commented on one tests, but similar changes can be made to the others.

I will merge tomorrow to give you time to address those if you want.

[pratik0316]

Thanks for the super fast review @wjones127 🙌

trying to adress the suggestions shortly

[codecov]

Codecov Report

❌ Patch coverage is 99.28058% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/write/merge_insert.rs	99.24%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!