Problem
When merge_columns_df falls back to the slow path and the input DataFrame covers only some fragments, the commit succeeds but the new dataset version contains only the touched fragments. Untouched fragments disappear from the manifest, silently reducing the visible row and fragment counts.
This affects both row-address merges with filtered input and business-key merges. It is independent of the execution runner or namespace support.
Root cause
The slow path collects only the FragmentMetadata returned by fragment.merge(...) and passes that list to LanceOperation.Merge. The operation expects the complete fragment set for the new manifest; it does not automatically inherit omitted fragments.
The fast path already preserves untouched fragments by appending their existing metadata before committing, but the slow path does not.
Expected behavior
A partial merge should:
- preserve every existing row and fragment;
- populate the new columns for matched rows;
- leave the new columns NULL for unmatched rows, including rows in entirely untouched fragments;
- create exactly one new dataset version.
Proposed fix
Before constructing LanceOperation.Merge in the slow path, collect the IDs of rewritten fragments and append metadata for all untouched fragments from the driver dataset snapshot. Keep read_version set to the planned dataset version so concurrent changes fail with a conflict instead of mixing snapshots.
The fragment-preservation logic should be shared with the fast path to prevent the two commit paths from drifting.
Regression test
Create at least four fragments, merge data covering only part of two fragments, and assert:
- row count is unchanged;
- fragment count is unchanged;
- matched rows receive values;
- unmatched rows in touched fragments receive NULL;
- rows in untouched fragments remain visible and receive NULL;
- the dataset version increments once.
Problem
When merge_columns_df falls back to the slow path and the input DataFrame covers only some fragments, the commit succeeds but the new dataset version contains only the touched fragments. Untouched fragments disappear from the manifest, silently reducing the visible row and fragment counts.
This affects both row-address merges with filtered input and business-key merges. It is independent of the execution runner or namespace support.
Root cause
The slow path collects only the FragmentMetadata returned by fragment.merge(...) and passes that list to LanceOperation.Merge. The operation expects the complete fragment set for the new manifest; it does not automatically inherit omitted fragments.
The fast path already preserves untouched fragments by appending their existing metadata before committing, but the slow path does not.
Expected behavior
A partial merge should:
Proposed fix
Before constructing LanceOperation.Merge in the slow path, collect the IDs of rewritten fragments and append metadata for all untouched fragments from the driver dataset snapshot. Keep read_version set to the planned dataset version so concurrent changes fail with a conflict instead of mixing snapshots.
The fragment-preservation logic should be shared with the fast path to prevent the two commit paths from drifting.
Regression test
Create at least four fragments, merge data covering only part of two fragments, and assert: