`merge_insert` can silently skip updates when a stale scalar index returns a missing row address

[mmatczuk]

Description

A stale scalar index can return a row address that is no longer present in the source batch. During an in-place merge_insert update, the reconciliation loop matches the batch's _rowaddr column against the row addresses returned by index-assisted matching. The code assumes that every updated row address appears, in order, in the original batch. A stale index breaks this invariant.

At this point, the invariant is enforced only by a debug_assert! in merge_insert.rs:

.map(|(original_offset, row_addr)| {
    match updated_row_addr_iter.peek() {
        Some((updated_row_addr, _)) if *updated_row_addr == *row_addr => {
            updated_row_addr_iter.next().unwrap().1
        }
        // If we have passed the next updated row address, something went wrong.
        Some((updated_row_addr, _)) => {
            debug_assert!(
                *updated_row_addr > *row_addr,
                "Got updated row address that is not in the original batch"
            );
            (0, original_offset) // fallback used when debug assertions are disabled
        }
        _ => (0, original_offset),
    }
})

The result depends on whether debug assertions are enabled:

• Builds with debug assertions enabled: the code panics with Got updated row address that is not in the original batch.
• Release builds: the assertion is compiled out, and the branch returns (0, original_offset). This selects the original row from source_batches[0] instead of an updated row, so the requested update can be silently skipped. The merge can therefore complete successfully with incorrect data.

This has the same stale-scalar-index root cause as the sibling fragment-does-not-exist-style failures, but surfaces during update-address reconciliation rather than during the take itself.

Status

We have a reproducer that fails on v9.1.0-beta.5, but it depends on our dataset_ops fuzzer and we were unable to extract a simple standalone case. The reproducer stopped failing after #7672, but the debug_assert! and release fallback are still present in v9.1.0-beta.8.

Lance version

• Affected: v9.1.0-beta.5
• Exact input verified fixed on v9.1.0-beta.8, but the guard is still present.

Language binding

Rust