Blob v2: filtering on a blob field panics in Field::apply_projection instead of returning a count/error

[justinrmiller]

Summary

Filtering on a Blob v2 (lance.blob.v2) extension field triggers a Rust panic instead of returning a count or a normal error. In Python this surfaces as pyo3_runtime.PanicException.

ds.count_rows(filter="image.image_bytes IS NOT NULL")

where image_bytes is a Blob v2 field nested under a struct:

image: struct<image_bytes: extension<lance.blob.v2>, error: string>

The panic is the apply_projection assertion in rust/lance-core/src/datatypes/field.rs:

assertion failed: !children.is_empty() || !projection.contains_field_id(self.id) || self.children.is_empty()

Even if filtering on Blob v2 fields is intentionally unsupported, Lance should return a normal "unsupported predicate" error, never panic.

Root cause

The predicate planner projects the referenced column (image.image_bytes) and then folds those filter columns back into the scan projection. That round-trip goes through BlobHandling::unload_if_needed → Field::unloaded_mut, which replaces the Blob v2 struct's real children (data, uri, …) with the blob description children from BLOB_V2_DESC_LANCE_FIELD.

Those description children are built via Field::try_from(&ArrowField) without any LANCE_FIELD_ID_KEY, so every description child gets field id -1 (field.rs try_from, None => -1).

Projection::union_schema then records only fields with id >= 0, so it keeps the parent blob field id but drops all of the (now id -1) child ids. When that projection is re-applied against the real dataset schema in Field::apply_projection:

• all of the blob field's real children are projected away → children.is_empty()
• the parent blob field id is still selected → contains_field_id(self.id)
• the blob field is nested → !self.children.is_empty()

…all three conditions of the assertion are violated → panic.

Reproduction

This repo's current write path only detects top-level Blob v2 columns (rust/lance/src/dataset/blob.rs blob_v2_cols, rust/lance/src/dataset/write.rs), so a nested Blob v2 dataset can't be written here to repro end-to-end. The faulty logic is in the projection layer, though, and is fully version-independent. The following lance-core unit test panics deterministically (both for nested and top-level Blob v2):

// rust/lance-core/src/datatypes/schema.rs (tests)
#[test]
fn repro_nested_blob_v2_projection_panic() {
    use lance_arrow::{ARROW_EXT_NAME_KEY, BLOB_V2_EXT_NAME};

    // image: struct<image_bytes: extension<lance.blob.v2>, error: utf8>
    let blob_meta: HashMap<String, String> =
        [(ARROW_EXT_NAME_KEY.to_string(), BLOB_V2_EXT_NAME.to_string())]
            .into_iter()
            .collect();
    let image_bytes = ArrowField::new(
        "image_bytes",
        ArrowDataType::Struct(ArrowFields::from(vec![
            ArrowField::new("data", ArrowDataType::LargeBinary, true),
            ArrowField::new("uri", ArrowDataType::Utf8, true),
        ])),
        true,
    )
    .with_metadata(blob_meta);
    let error = ArrowField::new("error", ArrowDataType::Utf8, true);
    let image = ArrowField::new(
        "image",
        ArrowDataType::Struct(ArrowFields::from(vec![image_bytes, error])),
        true,
    );

    let mut schema = Schema::try_from(&ArrowSchema::new(vec![image])).unwrap();
    schema.set_field_id(None);
    let base = Arc::new(schema);

    // Mirrors Scanner::filtered_projection for `image.image_bytes IS NOT NULL`
    let filter_schema = Projection::empty(base.clone())
        .union_column("image.image_bytes", OnMissing::Error)
        .unwrap()
        .into_schema();
    let scan_projection = Projection::empty(base.clone()).union_schema(&filter_schema);
    let _read_schema = scan_projection.into_schema(); // panics in Field::apply_projection
}

Output:

thread '...repro_nested_blob_v2_projection_panic' panicked at rust/lance-core/src/datatypes/field.rs:277:9:
assertion failed: !children.is_empty() || !projection.contains_field_id(self.id) || self.children.is_empty()

Environment (as reported)

• Lance version: 9.0.0-beta.15, Python 3.12, storage version 2.2
• Verified against main (repo at 5.1.0-beta.3); the assertion is at rust/lance-core/src/datatypes/field.rs:277 here (reporter saw it at field.rs:302 in their build).

Expected vs actual

• Expected: a row count, or a regular unsupported-operation error.
• Actual: Rust panic, surfaced as pyo3_runtime.PanicException in Python.

Notes / possible fixes

• The assertion in Field::apply_projection should not be reachable from user-triggered filter planning — a projection that selects a parent but none of its children should degrade to a clean error rather than assert!.
• The deeper issue is that union_schema silently drops the blob's description children because they carry field id -1; the filter-column round-trip through unload_if_needed loses the child selections. Either the blob field should not be unloaded while building the filter projection, or apply_projection should treat a fully-unloaded blob field consistently.
• This affects both nested and top-level Blob v2 fields; the reporter only exercised the nested case.

[mansiverma897993]

@wjones127 is this issue is resloved can you assign t his issue to me !!