Skip to content

Slow-path partial-fragment merge silently drops untouched fragments #55

Description

@FANNG1

Problem

When merge_columns_df falls back to the slow path and the input DataFrame covers only some fragments, the commit succeeds but the new dataset version contains only the touched fragments. Untouched fragments disappear from the manifest, silently reducing the visible row and fragment counts.

This affects both row-address merges with filtered input and business-key merges. It is independent of the execution runner or namespace support.

Root cause

The slow path collects only the FragmentMetadata returned by fragment.merge(...) and passes that list to LanceOperation.Merge. The operation expects the complete fragment set for the new manifest; it does not automatically inherit omitted fragments.

The fast path already preserves untouched fragments by appending their existing metadata before committing, but the slow path does not.

Expected behavior

A partial merge should:

  • preserve every existing row and fragment;
  • populate the new columns for matched rows;
  • leave the new columns NULL for unmatched rows, including rows in entirely untouched fragments;
  • create exactly one new dataset version.

Proposed fix

Before constructing LanceOperation.Merge in the slow path, collect the IDs of rewritten fragments and append metadata for all untouched fragments from the driver dataset snapshot. Keep read_version set to the planned dataset version so concurrent changes fail with a conflict instead of mixing snapshots.

The fragment-preservation logic should be shared with the fast path to prevent the two commit paths from drifting.

Regression test

Create at least four fragments, merge data covering only part of two fragments, and assert:

  • row count is unchanged;
  • fragment count is unchanged;
  • matched rows receive values;
  • unmatched rows in touched fragments receive NULL;
  • rows in untouched fragments remain visible and receive NULL;
  • the dataset version increments once.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions