Motivation
Lance can build one shared BM25 scorer when a query is executed over all committed FTS index segments in one process. A distributed query engine may instead assign different segment subsets to independent workers.
If every worker derives BM25 statistics only from its local segments, document counts, average document lengths, and term document frequencies differ between workers. Their scores are not comparable, and merging worker-local Top-K results cannot recover globally consistent ranking.
Apache Doris is one downstream use case:
The Doris FE Java layer pins one Lance dataset version, resolves the complete committed FTS segment set, and collects query-bound global BM25 statistics. It places the opaque payload in TFullTextSearchParams.global_statistics, which is delivered directly to each BE through the existing Thrift scan plan. Each BE will use lance-c to apply the same prepared statistics while searching its assigned segment subset.
V1 contract
V1 deliberately represents one BM25 corpus:
- one dataset snapshot;
- one logical FTS index, indexed column, and document granularity;
- the exact committed segment UUID set;
- corpus document count and total token count;
- document frequency for every prepared term;
- canonically ordered prepared query leaves, including final fuzzy vocabulary and original token positions.
Every scoring leaf must resolve to that single corpus. Producers must reject cross-column or otherwise cross-corpus queries.
The protobuf is an opaque payload for a trusted planning flow. V1 does not carry enough source-query information for a consumer to independently prove query identity, so the planner must attach it only to the same query used to produce it. Consumers still validate the schema version, dataset snapshot, logical index, column, document granularity, segment UUIDs, required fields, and canonical ordering.
API work
The two PRs intentionally separate the execution wire schema from its implementation.
Follow-up work
- Add a Rust/lance-c consumer that validates the payload and installs the prepared FTS scoring context on a scanner restricted to an explicit segment subset.
- Integrate the producer and consumer in Doris using the existing Thrift field.
- Add end-to-end multi-segment tests showing that distributed Top-K ranking matches one full-index execution.
- Define the behavior for unindexed fragments and mixed indexed/unindexed execution.
Completion criteria
- Every worker using the same payload produces scores comparable to a single full-segment execution.
- Fuzzy queries replay the globally prepared vocabulary without recomputing it from a worker-local subset.
- Cross-corpus queries are rejected explicitly in V1.
- Dataset, index, column, granularity, and segment-set mismatches fail validation.
Motivation
Lance can build one shared BM25 scorer when a query is executed over all committed FTS index segments in one process. A distributed query engine may instead assign different segment subsets to independent workers.
If every worker derives BM25 statistics only from its local segments, document counts, average document lengths, and term document frequencies differ between workers. Their scores are not comparable, and merging worker-local Top-K results cannot recover globally consistent ranking.
Apache Doris is one downstream use case:
The Doris FE Java layer pins one Lance dataset version, resolves the complete committed FTS segment set, and collects query-bound global BM25 statistics. It places the opaque payload in
TFullTextSearchParams.global_statistics, which is delivered directly to each BE through the existing Thrift scan plan. Each BE will use lance-c to apply the same prepared statistics while searching its assigned segment subset.V1 contract
V1 deliberately represents one BM25 corpus:
Every scoring leaf must resolve to that single corpus. Producers must reject cross-column or otherwise cross-corpus queries.
The protobuf is an opaque payload for a trusted planning flow. V1 does not carry enough source-query information for a consumer to independently prove query identity, so the planner must attach it only to the same query used to produce it. Consumers still validate the schema version, dataset snapshot, logical index, column, document granularity, segment UUIDs, required fields, and canonical ordering.
API work
The two PRs intentionally separate the execution wire schema from its implementation.
Follow-up work
Completion criteria