[VOTE] Persist effective FTS tokenizer configuration for code search

[Xuanwo]

• Reference PR: feat: add code analyzer for FTS #7681

Summary

This proposal extends InvertedIndexDetails so an FTS index persists the effective tokenizer configuration required to reconstruct its indexed term space.

Analyzer profiles such as code remain input-time presets. They are expanded into a document tokenizer, lexical tokenizer, and concrete tokenizer options before index details are persisted.

Problem

FTS queries, index updates, and segment merges must use the same tokenization behavior that was used when the index was created. Persisting only base_tokenizer is insufficient for code search because the resulting term space also depends on:

• How searchable text is extracted from the stored value.
• Whether identifiers are split into subwords.
• Whether letter/number boundaries are split.
• Whether the original identifier is preserved.
• Whether operators are indexed.

If these values are reconstructed from current defaults instead of persisted values, reopening or merging an index can produce terms that do not match the existing postings.

Proposed Design

Extend InvertedIndexDetails with the following fields:

// Document-level text extraction, such as "text" or "json".
optional string lance_tokenizer = 12;

// Code tokenizer options.
bool split_identifiers = 13;
optional bool split_on_numerics = 14;
optional bool preserve_original = 15;
bool index_operators = 16;

lance_tokenizer and base_tokenizer represent separate stages:

• lance_tokenizer extracts searchable text from the stored value.
• base_tokenizer performs lexical tokenization on that text.

This allows configurations such as JSON documents containing code to be represented as:

lance_tokenizer = "json"
base_tokenizer = "code"

Analyzer profiles are not persisted. For example, analyzer="code" is resolved into base_tokenizer="code" and concrete values for stemming, stop-word removal, identifier splitting, and other tokenizer options.

The persisted details therefore describe effective behavior rather than the preset used to produce it.

split_on_numerics and preserve_original use proto presence because their code-tokenizer defaults are true. An absent value must be distinguishable from an explicit false. split_identifiers and index_operators use regular booleans because their defaults are false.

The format maintains these invariants:

• Query-time tokenization matches index-time tokenization.
• All segments in one logical FTS index have equal normalized tokenizer details.
• Analyzer profile names do not participate in compatibility checks.
• Changes to profile defaults do not reinterpret existing indexes.

Compatibility and Migration

Existing indexes do not contain the new fields.

When lance_tokenizer is absent, readers infer it from the indexed column type and canonicalize the result before comparing segments.

When code-specific optional fields are absent, readers derive their defaults from base_tokenizer. Existing text indexes therefore retain their current behavior.

Older readers ignore the new protobuf fields. New readers can open both old and new indexes. New writers persist the resolved effective configuration so future reads, updates, and segment merges do not depend on client-side defaults.

[jackye1995]

+1

[prrao87]

+1 (binding)

[wjones127]

+1 (binding) for the proposal in the PR.