bug: address-domain scalar index references dropped fragments under stable row ids

[wjones127]

Under stable row ids, a scalar index whose results live in the address domain keeps referring to fragments that a later maintenance operation replaced. Reading through the index then fails hard in translate_addr_treemap_to_row_ids, which requires every referenced fragment to still be present:

Encountered internal error. Please file a bug report at https://github.com/lance-format/lance/issues.
fragment 0 referenced by an address-domain index result was not found in the dataset
  at rust/lance/src/dataset/rowids.rs:106

This makes a filtered scan unusable after routine maintenance. It is stable-row-id specific — the same sequence without enable_stable_row_ids succeeds.

Reproduction

Zone map index followed by compaction:

// rust/lance/tests/repro_addr_domain_index.rs
use std::sync::Arc;

use arrow_array::{Int32Array, RecordBatch, RecordBatchIterator};
use arrow_schema::{DataType, Field, Schema};
use futures::TryStreamExt;

use lance::dataset::optimize::{compact_files, CompactionOptions};
use lance::dataset::{WriteMode, WriteParams};
use lance::index::DatasetIndexExt;
use lance::session::Session;
use lance::Dataset;
use lance_index::scalar::{BuiltinIndexType, ScalarIndexParams};
use lance_index::IndexType;

fn schema() -> Arc<Schema> {
    Arc::new(Schema::new(vec![Field::new("i", DataType::Int32, false)]))
}

fn reader(range: std::ops::Range<i32>) -> impl arrow_array::RecordBatchReader + Send + 'static {
    let batch = RecordBatch::try_new(
        schema(),
        vec![Arc::new(Int32Array::from_iter_values(range))],
    )
    .unwrap();
    RecordBatchIterator::new(vec![Ok(batch)], schema())
}

#[tokio::test]
async fn repro_zonemap_compaction_references_dropped_fragment() {
    let dir = tempfile::tempdir().unwrap();
    let uri = dir.path().to_str().unwrap();
    let session = Arc::new(Session::new(
        512 * 1024 * 1024,
        512 * 1024 * 1024,
        Default::default(),
    ));
    let params = WriteParams {
        enable_stable_row_ids: true,
        session: Some(session.clone()),
        ..Default::default()
    };

    // Two fragments, so compaction has something to merge.
    Dataset::write(reader(0..100), uri, Some(params.clone()))
        .await
        .unwrap();
    let mut ds = Dataset::write(
        reader(100..200),
        uri,
        Some(WriteParams {
            mode: WriteMode::Append,
            ..params
        }),
    )
    .await
    .unwrap();

    ds.create_index(
        &["i"],
        IndexType::ZoneMap,
        None,
        &ScalarIndexParams::for_builtin(BuiltinIndexType::ZoneMap),
        false,
    )
    .await
    .unwrap();

    compact_files(&mut ds, CompactionOptions::default(), None)
        .await
        .unwrap();

    // The index still covers the pre-compaction fragment ids. Reading through
    // it must not surface an internal error.
    let mut scanner = ds.scan();
    scanner.filter("i > 0").unwrap();
    let batches: Vec<RecordBatch> = scanner
        .try_into_stream()
        .await
        .expect("plan")
        .try_collect()
        .await
        .expect("filtered scan through the zone map index must succeed");

    let rows: usize = batches.iter().map(|b| b.num_rows()).sum();
    assert_eq!(rows, 199, "i > 0 matches 199 of 200 rows");
}

cargo test -p lance --test repro_addr_domain_index

Actual, on e4288373a:

thread 'repro_zonemap_compaction_references_dropped_fragment' panicked at ...:
filtered scan through the zone map index must succeed: Internal {
  message: "fragment 0 referenced by an address-domain index result was not found in the dataset",
  location: Location { file: "rust/lance/src/dataset/rowids.rs", line: 106, column: 13 } }

Dropping enable_stable_row_ids: true makes the same sequence pass.

A second trigger, same error site

A bloom filter index followed by an update reaches the identical error, reporting fragment 1 instead. Found by our model-based test suite as the sequence:

write 100 rows -> write 400 rows -> create bloom filter index on int_col
  -> update int_col = -1 where category = 'A'

which surfaces as a missing index file because the index is unreadable afterwards:

Invariant violated: All index files exist
  index: int_col_idx
  missing file: _indices/28492248-ab0b-40ee-a5e3-82ea105ba762:
    Encountered internal error ... fragment 1 referenced by an address-domain
    index result was not found in the dataset, rust/lance/src/dataset/rowids.rs:120:13

I have not proven these two share a single fix, so they may need to be split — but both are the same class (an address-domain index result outliving the fragments it addresses) and both land on the same error site, so I have filed them together.

Expected

Index results should be reconciled against the live fragment set before translation — stale fragment references pruned rather than treated as an internal error — or the index should be invalidated/remapped by the operation that drops the fragments.

Environment

• Lance e4288373a
• Requires enable_stable_row_ids: true
• Affects ZoneMap (after compaction) and BloomFilter (after update)