Retrieval and Indexing

Retrieval and Indexing

Language: English · 简体中文

LWC uses deterministic lexical retrieval rather than embeddings. Every result can be traced to normalized terms, field matches, optional graph evidence, and audited adjustments.

Read this page when tuning retrieval, interpreting --explain, changing tokenization, or deciding whether graph state should affect ranking.

Retrieval pipeline

query
  -> normalize and deduplicate terms
  -> FTS5 document or span candidates
  -> mode, kind, and source-summary filters
  -> lexical field contributions
  -> audited weight and exact-query feedback
  -> bounded Wiki-link graph reranking
  -> mixed-granularity fusion and optional grouping
  -> deterministic type/rank/identifier order

No stage calls an external model. The same store, query, options, and adjustment state produce the same order.

Tokenization

LWC lowercases alphanumeric terms, removes a small fixed English/Chinese stop-word list, and preserves stable UTF-8 positions.

For contiguous CJK text:

• query tokenization emits adjacent bigrams;
• a single CJK character remains searchable;
• index tokenization also stores unigrams to preserve recall for one-character queries.

This dictionary-free design handles product names, code names, mixed languages, and new vocabulary without a segmentation service. Query duplicates are removed in first-seen order.

Queries containing only punctuation or stop words return no candidates. Document and span search accept at most 64 searchable terms.

Index fields

Document FTS uses four indexed fields with current BM25 weights:

Field	Weight
Title terms	8
Path terms	6
Summary terms	4
Body terms	1

Page and Source rows share one index. Metadata columns such as document type and identifier are unindexed and used for filtering and joining canonical records.

Span FTS indexes title, path, and exact passage or sentence body terms. Span text itself remains in the canonical Page or Source; byte ranges retrieve it without storing another mutable copy.

Document, passage, and sentence granularity

lwc --scope project search "checkpoint graph recovery" --granularity document
lwc --scope project search "checkpoint graph recovery" --granularity passage
lwc --scope project search "checkpoint graph recovery" --granularity sentence
lwc --scope project search "checkpoint graph recovery" --granularity all --group-by document

Markdown-aware segmentation recognizes headings, paragraphs, list items, block quotes, table rows, code blocks, and HTML. Sentence boundaries use Unicode segmentation; code blocks use non-empty code lines.

Each locator includes document identity, parent, ordinal, UTF-8 byte range, fingerprint, and segmenter version. A modified document invalidates the old locator instead of returning similar but unproven text.

Lexical ranking

SQLite BM25 produces the base rank; lower is better. LWC adds explainable contributions:

• exact or phrase-like title matches receive the strongest promotion;
• path matches receive a smaller promotion;
• generic navigation/index documents receive a penalty when their generic role is unrelated to the query;
• stable type and identifier keys break ties.

The current contribution constants are title -32, path -16, and generic marker +8, multiplied by their normalized signals. These constants are implementation details exposed by --explain, not a promise that users should hand-compute ranks.

Page-first source behavior

In auto and all target modes, maintained Pages sort before raw Sources. In auto, a raw Source is suppressed when a kind=source Page already summarizes that Source.

This preference keeps synthesis visible while preserving a route back to immutable evidence. Use --target source when raw Source matching is the explicit task.

Audited adjustments

Manual weights accept -2, -1, 1, or 2. Positive values promote and negative values demote an existing lexical candidate. Exact-query feedback stores relevant or irrelevant against the SHA-256 fingerprint of normalized terms.

lwc --scope project weight set page recovery-rules 2 \
  --reason "Core recovery guidance" \
  --provenance user-provided
lwc --scope project weight feedback page recovery-rules \
  --query "checkpoint graph recovery" \
  --signal relevant \
  --reason "Verified expected answer" \
  --provenance agent-observed

Current maximum contributions are -2 for positive manual weight and -1.5 for relevant feedback; opposite signals demote by the same magnitude. A user-provided row takes precedence over agent-observed for the same target and adjustment class.

Adjustments never create candidates. A document must first match lexical terms. Feedback applies only to the exact normalized-query fingerprint and does not silently transfer to paraphrases.

Wiki-link graph reranking

LWC takes at most the first three lexical Page candidates as graph seeds. Canonical Wiki links and shared Source citations provide bounded related-page evidence:

• matching structural evidence can promote a Page by at most 0.25 rank units;
• generic high-outdegree navigation Pages can receive a hub penalty up to 4;
• later seeds contribute with a position discount.

This reranking reads canonical SQLite Page relationships. It does not require the external document-graph engine and cannot rescue a nonmatching Page from outside the lexical candidate set.

Mixed-granularity fusion

--granularity all combines documents, passages, and sentences with reciprocal-rank-style fusion. Sentence and passage matches receive small priors, and each granularity contributes prior / (60 + position).

With --group-by document, LWC groups up to the three best matches for each document and combines them with weights 1, 0.5, and 0.25. The response retains the selected match locators so callers can inspect exact evidence.

Explanation

lwc --scope project search "checkpoint graph recovery" --limit 10 --explain

explanation reports:

• BM25 base rank;
• title, path, generic, manual, feedback, graph-match, and graph-hub signals;
• each numeric contribution;
• final rank;
• graph seed evidence when present.

Use it before changing weights. A disappointing result may be caused by missing terminology, a generic title, stale content, raw-source target mode, or absent lexical candidates rather than poor constants.

Reindexing and consistency

Page and Source mutations update their FTS and span rows transactionally. lint detects missing, duplicate, or orphan document index rows.

If projection state is damaged, run maintenance Work and inspect its result:

lwc --scope project maintenance reindex
lwc --scope project work watch <work-id>
lwc --scope project lint

Reindexing is a repair operation, not routine search preparation. Normal reads never rebuild indexes implicitly.

Retrieval acceptance

For durable knowledge changes, predeclare at least two checks:

1. the exact expected wording;
2. a natural paraphrase using different surface terms.

Require the intended Page in the top five for both, inspect Source citations, and repeat against live state after changeset commit. Do not tune a global weight to hide one poorly written Page.

Next: Performance and scalability