Reading and Filters

Stats

A table created with WriteOptions { stats: true, .. } stores the min and the max of each variable per row. Each variable gets one stats column, __stats__:<variable>, with min and max. Null and NaN values do not count. Numeric, string, timestamp, date, time and duration variables get stats. An append writes stats when the table has them.

A writer adds a variable and its stats column together. So a fragment holds only the stats columns of its own rows, and a stats filter also works on older fragments without the variable. In Lance SQL, the column name needs backticks: `__stats__:sst`.max.

lance_nd::datafusion is the DataFusion version of Lance. Use it to build filters.

Pruning

A scan with filter_expr uses the stats to skip the rows that surely have no matching value. lance-nd changes each part of the filter that reads <variable>.values into a test on the min and the max of the row:

Filter on the values	Filter on the stats
v = x	min <= x AND max >= x
v < x, v <= x	min < x, min <= x
v > x, v >= x	max > x, max >= x
v BETWEEN a AND b	max >= a AND min <= b
v IN (x, y)	The = test for each value, joined with OR

Then Lance reads only the rows that pass the stats test. For example, with the filter lat.values BETWEEN 17.5 AND 25.4:

Row	lat min	lat max	Result
0	-90	-80	skipped
1	17	30	kept
2	absent	absent	skipped: the row does not have lat
3	null	null	kept: no value counts, for example all NaN

• A kept row comes back whole. It can hold values outside the filter.
• A value part without a stats form keeps all rows, for example NOT, != or t.values * 2 > 10.
• A table without stats keeps all rows for a value part.

Filters

A filter for filter_expr uses the Lance table columns:

use lance_nd::datafusion::functions::core::expr_fn::get_field;
use lance_nd::datafusion::prelude::{col, lit};

let lat = get_field(col("lat"), "values");
let units = get_field(get_field(col("sst"), "attrs"), "units");
let id = get_field(col("__global_attrs__"), "id");
let rows = scan(&table)
    .filter_expr(lat.between(lit(17.5), lit(25.4)).and(units.eq(lit("K")).or(id.eq(lit("a")))))
    .try_into_stream()
    .await?;

lance-nd does not check or change the column names. A wrong name gives an error in the scan. In an Expr, write a nested column such as sst.values with get_field: Lance does not resolve col("sst.values").

A part that does not read values, for example on __global_attrs__.id or sst.attrs.units, filters the rows exactly, so it stays as it is. Parts combine with these rules:

• AND: a side without a stats form goes away.
• OR: both sides need a form, or the full OR keeps all rows.
• NOT: only over parts that do not read values.

Use one filter API for each case:

Filter	API
Lance SQL text	filter
DataFusion Expr on the table columns, with values as stats	filter_expr
DataFusion Expr on the Lance table columns, with no change	lance_filter

filter_expr and lance_filter can be used together. A row must then match both. filter cannot be used with one of them.

A scan with a filter first reads only the filter columns. Then it reads the projected variables of the matching rows only. So a filter that matches few rows reads little data. A filter that matches all rows is a little slower than a scan without a filter. NdScanner::explain_plan shows the reads.