Full scan fails with an out-of-range projection after adding a column and compacting

[mmatczuk]

Description

A full scan can fail after add_columns and compact_files with an internally generated column
index that is one past the end of a data file:

The projection specified the column index 11 but there are only 11 columns in the file

The scan requests every column by name, so the out-of-range physical column index is produced
inside Lance. The error comes from the v2 file reader's
projection bounds check.

The failing dataset uses storage version 2.3 with stable row IDs disabled. Before the failure, it
undergoes one append, seven literal updates, and two merge-insert upserts. It then adds an all-null
Dictionary<Int32, Utf8> column, compacts the files, and scans all columns. The existing columns
are Struct<Time32Ms, Time32Ms> values.

A short public-API reproducer has not been found. add_columns followed by compact_files and a
full scan succeeds in the simpler cases tested across storage versions 2.1 and 2.3, with 1, 3, or
10 flat value columns and max_rows_per_file set to 2 or 1024. This suggests that the preceding
fragment rewrites and/or nested physical column layout are required to trigger the inconsistency.

Expected behavior

A full scan after adding a column and compacting should return the dataset's rows. Projection names
should be resolved against each data file's actual column layout, including files written before
the column was added.

Lance version

v10.1.0-beta.2 (94cca93d13b1c731afd6660d46ad2f8d4b4345c4)

Language binding

Rust

Environment

Fedora Linux 44, x86_64, in-memory object store, rustc 1.97.1

Logs / traceback

thread 'main' panicked at src/dataset_ops.rs:2323:29:
step 4: full scan: scan execution failed: Invalid user input: The projection specified the column
index 11 but there are only 11 columns in the file,
rust/lance-file/src/reader.rs:1341:28

[lance-gatefixer]

I examined main at 86d3bd1 but could not reproduce the reported out-of-range projection. With v2.3, stable row IDs disabled, five Struct<Time32Ms, Time32Ms> columns, and two-row files, a metadata-only all-null Dictionary<Int32, Utf8> addition fails earlier while compacting because the empty dictionary value array cannot be encoded; it never creates the reported post-compaction file. Adding the same all-null dictionary from a reader with one unused dictionary value compacts and scans successfully.

Could you attach the failing dataset files, or provide the exact initial schema/data plus the seven update predicates/expressions and two merge-insert source batches/options? That is the minimum evidence needed to identify which rewrite emits column index 11.