Skip to content

feat: route partial-schema merge_insert through the v2 write path #6472

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wjones127 merged 5 commits into from
May 12, 2026
+683 −102
Merged

feat: route partial-schema merge_insert through the v2 write path #6472

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
04cbcb3
feat: route partial-schema merge_insert through the v2 write path
Apr 10, 2026
ce1dfe9
test: cover camelCase columns in partial-schema v2 merge_insert
Apr 11, 2026
b80f89b
fix: use object_store field directly after base-aware API change
May 4, 2026
41de6d2
Merge branch 'main' into feat/partial-schema-v2-merge-insert-6442
wombatu-kun May 7, 2026
b34ea77
Merge branch 'main' into feat/partial-schema-v2-merge-insert-6442
wjones127 May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 8 additions & 15 deletions python/python/tests/test_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2269,7 +2269,6 @@ def test_merge_insert_subcols(tmp_path: Path):
dataset = lance.write_dataset(
initial_data, tmp_path / "dataset", max_rows_per_file=5
)
original_fragments = dataset.get_fragments()

new_values = pa.table(
{
Expand All @@ -2279,6 +2278,12 @@ def test_merge_insert_subcols(tmp_path: Path):
)
(dataset.merge_insert("a").when_matched_update_all().execute(new_values))

# Partial-schema upserts run through the v2 `FullSchemaMergeInsertExec`
# path (see Rust issue #6442). We only assert the semantic outcome here
# — the exact fragment / file layout is an implementation detail of the
# v1 RewriteColumns optimization that no longer applies. Column `c`
# (not in the source) must retain its original values for the updated
# rows.
expected = pa.table(
{
"a": range(10),
Expand All @@ -2288,20 +2293,6 @@ def test_merge_insert_subcols(tmp_path: Path):
)
assert dataset.to_table().sort_by("a") == expected

# First fragment has new file
fragments = dataset.get_fragments()
assert fragments[0].fragment_id == original_fragments[0].fragment_id
assert fragments[1].fragment_id == original_fragments[1].fragment_id

assert len(fragments[0].data_files()) == 2
assert (
fragments[0].data_files()[0].path == original_fragments[0].data_files()[0].path
)
assert len(fragments[1].data_files()) == 1
assert str(fragments[1].data_files()[0]) == str(
original_fragments[1].data_files()[0]
)

new_values = pa.table(
{
"a": range(9, 12),
Expand All @@ -2316,6 +2307,8 @@ def test_merge_insert_subcols(tmp_path: Path):
)

assert dataset.count_rows() == 12
# Newly inserted rows (keys 10, 11) get NULL for column `c` because
# the source did not provide a value for it and `c` is nullable.
expected = pa.table(
{
"a": range(0, 12),
Expand Down
Loading
Loading