Lance FTS V3 Term Spaces

[Xuanwo]

Abstract

This document proposes introducing FTS V3: retain V2 default behaviors and the existing tokenizer parameter model, while adding minimal format capabilities to support future multiple term spaces.

V3 continues to use the current tokenizer, filters, scoring, and query semantics by default. The first goal of V3 is to keep the single-space path close to V2, while allowing the format to express multiple independent statistical spaces when needed.

Core concept:

term space:
  a set of terms that share the same statistical semantics and scoring semantics

The vast majority of existing tokenizers require only a single term space:

implicit term_spaces = ["primary"]

In the future, some analyzers may need multiple term spaces. For example, the same text may simultaneously produce word terms, subword terms, phonetic terms, stemmed terms, or analyzer-specific auxiliary terms. When these terms require independent doc length, avgdl, or df, they should go into different term spaces.

This document focuses on lexical full-text search. Cross-language semantic matching should continue to be handled by vector search or hybrid search.

Motivation

The current FTS model is:

document text
  -> tokenizer
  -> tokens
  -> one token dictionary
  -> one posting list space
  -> one document length
  -> one BM25 score

This model works well for single-tokenizer text. V3 takes this model as its default path.

V3 adds a narrower format capability: when an analyzer produces multiple term sets with different statistical semantics, the index format can express that “a single field produces terms through multiple analysis channels, and maintains independent BM25 statistics for those terms.”

Multiple term spaces are a general format capability, applicable to any language and any analyzer strategy.

For example, a future analyzer might produce:

word terms:
  high-confidence lexical words

subword terms:
  character ngrams, morphemes, compound splits, or other recall-oriented units

phonetic terms:
  sound-alike signatures

stemmed terms:
  normalized morphological forms

If all these terms are mixed into a single BM25 space:

• document length will be inflated by terms of different granularities;
• word terms, subword terms, and phonetic terms will naturally have equal weight;
• avgdl and df will blend different statistical semantics;
• low-confidence or high-frequency auxiliary terms may disturb primary ranking.

The design goal of FTS V3 is to allow these term spaces to be separated statistically when needed, while keeping the single-term-space path close to V2 in cost and behavior.

User-Visible Behavior

Default FTS behavior continues to align with current semantics:

ds.create_scalar_index("text", "INVERTED")

This still means the existing default tokenizer and filters.

Existing tokenizer options continue to work:

simple
whitespace
raw
ngram
jieba/...
lindera/...

In V3, these tokenizers remain single-space indexes:

implicit term_space_id = 0

In the future, new analyzers can be explicitly enabled, for example:

ds.create_scalar_index(
    "text",
    "INVERTED",
    base_tokenizer="icu+subword",
)

Whether multiple term spaces are produced is determined by the actual output of the analyzer; language, locale, or domain hints can participate in this decision as analyzer options.

Term Spaces

First-principles definition:

term space = a set of terms that share the same statistical semantics and scoring semantics

When two groups of terms require independent handling of the following statistics or semantics, use different term spaces:

doc length
average document length
document frequency
query participation behavior
position semantics

V3 must support single-space:

space 0: primary

Future versions may support multi-space:

space 0: primary
space 1: subword
space 2: phonetic
space 3: stemmed

These names are interpretive metadata. The format layer relies on stable term_space_id.

Example:

text: OpenAI 发布了新模型 v2

A Unicode baseline analyzer could produce only single-space terms:

space 0 / primary:
  openai
  发
  布
  了
  新
  模
  型
  v2

A more complex analyzer in the future could produce two spaces:

space 0 / primary:
  openai
  发布
  新模型
  v2

space 1 / subword:
  发
  布
  新
  模
  型
  发布
  新模
  模型

The same mechanism can also be used for other languages and strategies, such as compound splitting, morphology, phonetic search, or stemming/exact dual-space. Term spaces express the differences in statistical semantics between different term sets.

Parameters

V3 continues to use the existing InvertedIndexParams as the user-facing API for creating indexes.

Tokenizer/analyzer behavior is fixed at index creation; users switch tokenizers, filters, or analyzer behavior by dropping and recreating the index. V3 continues to encode this information using existing metadata mechanisms.

V3 should continue to store InvertedIndexParams in the existing params metadata. This already covers the information needed for query-time analyzer construction, such as:

base_tokenizer
language
filters
ngram params
with_position

V3 continues to use the existing configuration abstraction. The first version adds term-space layout metadata necessary for multi-space.

Suggested minimal additional metadata:

term_spaces: optional list<string>

Explanation:

• Single-space indexes use an implicit term_space_id = 0.
• Multi-space indexes write term_spaces, and the array index is the term_space_id.
• The strings in term_spaces are for debug/display; query execution uses the array index.
• The first version of multi-space scoring uses an unweighted BM25 sum.
• Locale-related capabilities can be introduced later as explicit tokenizer/analyzer options.
• Build-only fields such as memory limit and worker count remain build-time options.

The physical encoding of term_spaces uses the existing Lance index metadata mechanism.

On-Disk Format

V3 continues to use the existing file layout:

metadata.lance
part_<pid>_tokens.lance
part_<pid>_invert.lance
part_<pid>_docs.lance

The format supports multiple term spaces; single-space V3 maintains the V2 fast path.

metadata.lance

metadata.lance stores index-level metadata via schema metadata and deleted fragments via row data.

The FTS format version continues to be expressed using the existing index version mechanism.

Required schema metadata:

params = <existing persisted InvertedIndexParams>
partitions = <encoded partition ids>
token_set_format = "fst" or "arrow"
posting_tail_codec = "varint_delta_v1"
positions_layout = "shared_stream_v2" when positions are enabled
positions_codec = "packed_delta_v1" or later codec name
term_spaces = <encoded list<string>>, optional, only for multi-space

Logical example of term_spaces:

term_spaces:
  - primary
  - subword

Here primary and subword are display names. Format semantics are determined by array index:

primary -> term_space_id 0
subword -> term_space_id 1

Row data:

deleted_fragments: binary

Manifest details may retain a summary for planning; query execution reads persisted params and optional term_spaces from metadata.lance. Switching tokenizer/analyzer behavior is accomplished by dropping and recreating the index.

part__tokens.lance

The token dictionary is responsible for mapping (term_space_id, term) to a token id. V3 has two physical shapes.

Single-space V3:

implicit term_space_id = 0

It can use the same logical schema as the V2 FST token set:

_token_fst_bytes: LargeBinary
_token_next_id: UInt32
_token_total_length: UInt64

This way, single-space V3 is close to V2 in storage and query cost.

Multi-space V3:

term_spaces metadata is present

FST keys use a reserved binary term-space prefix:

encoded_key = term_space_id_u16_be || normalized_term

Example:

0x0000 || "database" -> token_id 42
0x0001 || "database" -> token_id 314

Token ids within a partition are globally unique. This is the format definition for V3 multi-space.

Therefore, invert.lance can continue to look up posting lists by token id only. The query analysis stage knows which term space a term belongs to and finds the corresponding token id via the space-aware key.

part__invert.lance

Posting lists continue to be indexed by token id:

token_id -> posting list

Logical schema:

_posting: List<LargeBinary>
_max_score: Float32
_length: UInt32
_compressed_position: LargeBinary, optional
_position_block_offset: List<UInt32>, optional

Posting payloads continue to store:

doc id
frequency
positions, optional
block max score

In V3, _max_score and per-block max scores are computed using the actual scoring model, ensuring that WAND pruning uses the correct score upper bound.

Because token ids are globally unique, invert.lance continues to index posting rows by token id only.

part__docs.lance

Document statistics are the key to V3 multi-term-spaces.

Single-space V3 can keep a V2-compatible schema:

_rowid: UInt64
_num_tokens: UInt32

At runtime this is interpreted as:

num_tokens_by_space[0] = _num_tokens

Multi-space V3 uses extensible per-space lengths:

_rowid: UInt64
_num_tokens_by_space: FixedSizeList<UInt32>[num_term_spaces]

Example:

term_spaces = ["primary", "subword"]

row text: OpenAI 发布了新模型 v2
_num_tokens_by_space = [4, 8]

These fields allow different term spaces to use independent BM25 statistics:

BM25(space 0) uses num_tokens_by_space[0]
BM25(space 1) uses num_tokens_by_space[1]

Independent doc length allows the statistics for auxiliary terms such as subword, phonetic, and stemmed terms to be maintained separately from primary terms.

Query and Scoring Semantics

V3 query flow:

load metadata.lance
read persisted params and optional term_spaces
build analyzer from persisted params
analyze query into terms grouped by term_space_id
look up token ids with space-aware encoding
load posting lists
run BM25 / WAND
return _rowid and _score

Single-space scoring is equivalent to existing BM25:

score = bm25(space 0)

Multi-space scoring:

score =
  sum over term spaces:
    bm25(space, doc_len(space), df(space), avgdl(space))

The first version of scoring uses an unweighted sum of per-space BM25.

Requirements:

• Each term space uses an independent document length;
• Each term space uses an independent average document length;
• PostingIterator must know the term space for each token;
• WAND upper bounds must use the same multi-space scoring model;
• In the first version of V3, phrase validation occurs within the same term space;
• The query analyzer sets an expansion limit for high-fan-out spaces to control the number of posting list scans.

For rows outside indexed fragments that must take the flat path, the flat path uses the same analyzer and the same term-space-aware scoring model, so that scores from indexed and unindexed results remain comparable.

Compatibility

V1/V2 indices remain readable.

For the same tokenizer and filters, V3 single-space should match V2 behavior. This is the first compatibility gate.

Append / update:

existing V1/V2 index:
  preserve current behavior unless rebuilt

existing V3 index:
  use persisted params and optional term_spaces from metadata.lance
  switch analysis behavior through drop/recreate

Optimize / compaction:

within the same physical FTS index:
  carry forward the same params, optional term_spaces, and codec metadata

across independently created FTS indices:
  combine through explicit rebuild

Default user behavior remains compatible with V2. In the future, V3 may become the default write format for new FTS indexes; the default analyzer will continue to be determined by existing tokenizer defaults.

Comparison with Industry Practices

Lucene’s Analyzer converts text into index terms and is a field-aware analysis strategy; Lucene also provides analyzer packages for ICU, Japanese, Chinese, phonetic, and others. This shows that analysis behavior is a core semantic that search systems need to record and reproduce. Lucene BM25 statistics such as average field length and idf are also computed per field.

Elasticsearch and OpenSearch multi-fields both allow “re-indexing the same field with different analyzers into different subfields.” This aligns with the motivation for term spaces: the same original text can have multiple analyzed representations. Elasticsearch/OpenSearch implement this through multiple Lucene fields; Lance V3 expresses the same capability within a single FTS index via term spaces.

Elasticsearch’s combined_fields uses a BM25F-like approach to score multiple text fields as a combined field, and the documentation notes that it requires participating fields to have the same search analyzer and that the clause count grows with fields × terms. The takeaway for Lance is that multiple term spaces need to control query fan-out, and scoring requires an explicit statistical model.

Typesense’s locale-specific search supports per-field locale and provides pre_segmented_query to ensure alignment between indexing and query tokenization. This supports two conclusions: first, analyzer consistency needs to be persisted; second, complex language or domain tokenizers require explicit tokenizer controls.

Meilisearch uses a multilingual tokenizer pipeline, and in its multilingual dataset guide recommends splitting indexes by language first; if a single index is used, it suggests using localized attributes/query locales to align document and query tokenization, and warns that performance and relevancy may degrade as languages mix and grow. This supports the direction of Lance V3: keep the V2 single-statistical-space behavior by default, and enable multi-space storage when the analyzer explicitly requires it.

Tantivy’s TextAnalyzer / token filter model is close to Lance’s current tokenizer design, indicating that a single token stream remains the default primary path. V3 continues this primary path and enables multi-space when the analyzer explicitly needs multiple statistical spaces.

References to official documentation:

• Lucene Analyzer
• Lucene BM25Similarity
• Elasticsearch multi-fields
• Elasticsearch combined_fields
• Elasticsearch similarity
• OpenSearch fields
• Typesense locale-specific search
• Meilisearch multilingual datasets
• Meilisearch tokenization
• Tantivy tokenizer