Improve chunking and search retrieval quality

[RobertLD]

Problem Statement

LibScope's current chunking and search pipelines have several limitations that reduce retrieval quality — the core value prop of a knowledge base. This issue tracks a set of targeted, incremental improvements informed by a codebase audit and independent technical review.

---

Current State

Chunking (src/core/indexing.ts)

• chunkContent() splits on markdown h1-h3 headings, hard-caps at 1500 chars
• Injects <!-- context: H1 > H2 --> breadcrumbs as HTML comments (noisy for embeddings)
• When maxChunkSize is hit, cuts at the current line — no paragraph awareness
• Chunk embeddings contain only chunk.content — document title, library, version, and topic are never embedded (biggest quality hole)
• No inter-chunk overlap (while chunkContentStreaming() has 10% overlap)

Search (src/core/search.ts)

• Sequential fallback: vector → FTS5 → LIKE (no hybrid fusion)
• FTS5 query uses OR logic ("React" OR "hooks") — extremely noisy for multi-word queries
• Title never factors into ranking
• Count query re-executes the full search wrapped in SELECT COUNT(*)
• No retrieval quality tests exist — all tests are purely functional

---

Proposed Improvements

Improvement 0: Retrieval Quality Benchmark & Regression Gate — P0 (FIRST)

Create a curated test corpus (~10-15 realistic docs across overlapping topics), ~20 test queries with ground-truth expected results, and metric functions (Recall@k, MRR). Establish baseline quality metrics for the current FTS5 implementation, then use those baselines as CI regression gates — any future change that degrades retrieval quality fails CI. Thresholds ratchet upward as improvements land.

Components:

• tests/fixtures/benchmark-corpus.ts — curated test documents
• tests/fixtures/benchmark-queries.ts — ground-truth query set
• tests/fixtures/benchmark-metrics.ts — Recall@k, MRR metric functions
• tests/benchmark/retrieval-quality.test.ts — benchmark test suite

Constraints: Runs against FTS5 only (test DB has no sqlite-vec). Uses MockEmbeddingProvider. This is acceptable — FTS5 is the production fallback path.

---

Improvement 1: Metadata Embedding — P0

Prepend structured document metadata (title, library, version, topic) to each chunk before computing its embedding. Store original chunk content without prefix in the DB.

const metadataPrefix = [
  `Title: ${input.title}`,
  input.library ? `Library: ${input.library}` : null,
].filter(Boolean).join("\n") + "\n\n";

const chunksForEmbedding = chunks.map(chunk => metadataPrefix + chunk);
const embeddings = await provider.embedBatch(chunksForEmbedding);

Impact: Fixes the biggest quality hole — metadata-based queries can't match semantically today.

---

Improvement 2: Chunk Overlap — P0

Add configurable overlap (~150 chars, ~10% of maxChunkSize) between consecutive chunks. Tail of chunk N is prepended to chunk N+1. Align chunkContentStreaming() to use the same configurable overlap.

Impact: Industry-standard RAG practice. Significantly improves recall for boundary-spanning queries.

---

Improvement 3: Replace HTML Comment Breadcrumbs — P1

Replace <!-- context: H1 > H2 --> with plain text prefix H1 > H2\n. HTML comments waste embedding token budget and dilute semantic signal.

---

Improvement 4: Title Boosting — P1

Boost result score when query terms appear in document title (case-insensitive, configurable factor ~1.5x). Include in scoreExplanation.boostFactors. Trivial to implement (~30 lines).

---

Improvement 5: FTS5 AND-by-Default — P1

Change FTS5 query from "w1" OR "w2" to "w1" "w2" (implicit AND). Support quoted phrase search. No auto-OR fallback — return zero results rather than confusing non-deterministic behavior.

---

Improvement 6: Hybrid Search (RRF) — P0-deferred

Run vector + FTS5 search, merge via Reciprocal Rank Fusion (score = Σ 1/(60 + rank)). Add searchMode option: hybrid/vector/keyword. Graceful fallback. Deferred until foundation is solid.

Testing strategy: RRF fusion function is pure (takes two ranked lists → merged list) — fully testable without sqlite-vec. Unit tests mock vector path.

---

Improvement 7: Lazy Count — P2

Make totalCount optional, add hasMore: boolean. Skip expensive double count query by default. Opt-in via countMode: 'exact'.

---

Improvement 8: Paragraph-Boundary Splitting — P3

When maxChunkSize is hit, scan backward for \n\n within last 200 chars. If found, split there. No code block tracking, no strategy enum. 80% of benefit for 20% of complexity.

---

Implementation Priority & Order

#	Improvement	Impact	Effort	Priority
0	Quality Benchmark & Regression Gate	🔴 High	Medium	P0 — FIRST
1	Metadata Embedding	🔴 High	Low	P0
2	Chunk Overlap	🔴 High	Low	P0
3	Text Breadcrumbs	🟠 Medium	Low	P1
4	Title Boosting	🟠 Medium	Low	P1
5	FTS5 AND-by-Default	🟠 Medium	Low	P1
6	Hybrid Search (RRF)	🔴 High	Medium	P0-deferred
7	Lazy Count	🟢 Low	Medium	P2
8	Paragraph Splitting	🟢 Low	Low	P3

Order: 0 → 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8

Notes

• Quality-first: Benchmark (0) is implemented first so every subsequent improvement is objectively measurable. Thresholds ratchet upward after each improvement.
• Re-indexing: Chunking improvements (1-3, 8) require existing docs to be re-indexed. Consider a chunking_version field for selective re-indexing.
• API compatibility: Improvement 7 changes SearchResponse shape — use feature flags / make totalCount optional.
• Testing: Hybrid search RRF fusion logic should be a pure function, testable without sqlite-vec.

[RobertLD]

Baseline Retrieval Quality — Benchmark Results

Ran a benchmark using the real production pipeline: LocalEmbeddingProvider (all-MiniLM-L6-v2, 384-dim) + sqlite-vec vector search + the full searchDocuments() code path.

Setup

• Corpus: 12 documents across distinct topics (React hooks, Express routing, PostgreSQL indexes, JWT auth, Docker Compose, TypeScript generics, Python venvs, Git branching, React Server Components, SQLite tuning, CSS Grid)
• Queries: 20 natural-language queries with ground-truth expected document titles
• Metrics: Recall@k (fraction of expected docs in top-k results), MRR (reciprocal rank of first relevant result)

Aggregate Results

Metric	Score
Recall@1	0.950
Recall@3	0.950
Recall@5	0.975
MRR	1.000

• Perfect hit rate: 20/20 queries found at least one expected doc in top 1
• Total failures (no expected doc in top 5): 0/20

What the Numbers Actually Mean

The scores look great, but this baseline has important limitations that inflate the numbers:

1. Small, well-separated corpus

12 documents on completely distinct topics. With real usage (hundreds of docs, overlapping domains), vector search has to make much harder discrimination decisions. This corpus doesn't stress-test that.

2. No per-document dedup in results

Results show the same document title repeated 4-5 times across top-5 (different chunks from the same doc). For example, query "API authentication with tokens" returns 5 chunks all from "JWT Authentication". This is correct but monopolizes result slots, which directly hurts multi-topic queries.

3. Multi-topic queries are the weak spot

• "how to create database indexes for full text search" — Expected [PostgreSQL Indexing, SQLite Tuning]. Got PostgreSQL in positions 1-3, SQLite only at position 4. Recall@3 = 0.50 because one doc monopolized slots.
• "how to verify JWT tokens in a Node.js middleware" — Expected [JWT Auth, Express Routing]. Got only JWT Auth chunks (all 5 slots). Recall@5 = 0.50 — Express middleware never surfaced.

4. Metadata not embedded (invisible to vector search)

The queries didn't test metadata-based retrieval (e.g., "show me react 18 docs") because the current chunking doesn't embed title/library/version into chunk vectors. This is the biggest real-world quality hole — a user searching by library name gets results based purely on content similarity, not metadata match.

5. FTS5 not tested

Vector search succeeded for all 20 queries, so the FTS5 fallback path was never exercised. FTS5 quality (with its current OR-logic) was not measured here.

Key Takeaways for the Improvement Plan

1. maxChunksPerDocument should default to something reasonable (2-3) — currently unlimited, same doc can fill all result slots
2. Metadata embedding (Improvement 1) won't show gains on this benchmark because queries are content-based — need metadata-specific queries to measure it
3. Chunk overlap (Improvement 2) matters most for boundary-spanning queries on long docs — this corpus has short docs where chunks are self-contained
4. The real benchmark needs: 50+ docs with overlapping topics, metadata-targeted queries, and maxChunksPerDocument dedup to surface the weaknesses the plan addresses
5. Current vector search is solid for the easy case — distinct topics, content-match queries. The improvements target the hard cases (metadata search, ambiguous multi-topic queries, FTS5 quality).

Benchmark Methodology

The benchmark script indexes the corpus using indexDocument(), queries using searchDocuments(), and measures Recall@k and MRR against ground-truth labels. It uses an in-memory SQLite DB with sqlite-vec loaded (the same pipeline production users hit). Full script was run as npx tsx tests/benchmark/baseline-retrieval.ts and cleaned up after capturing results.

[RobertLD]

Expanded Baseline Benchmark — 38 docs, 34 queries, 5 categories

Follow-up to the initial 12-doc benchmark. This test uses overlapping topics to stress-test discrimination and cross-topic retrieval.

Setup

• 38 documents across overlapping domains: 6 React docs, 4 Node.js/Express docs, 6 database docs, 4 auth/security docs, 5 DevOps docs, 4 language docs, 3 frontend docs, 4 architecture docs, 2 testing docs
• 34 queries in 5 categories: easy (10), cross-topic (8), metadata-targeted (5), adversarial (6), ambiguous (5)
• Real pipeline: LocalEmbeddingProvider (all-MiniLM-L6-v2, 384-dim) + sqlite-vec vector search
• limit=10, no maxChunksPerDocument (current default behavior)

---

Aggregate Metrics (raw, no dedup)

Metric	Score
Recall@1	0.824
Recall@3	0.850
Recall@5	0.909
Recall@10	0.983
MRR	0.985

Per-Category Breakdown

Category	Queries	Recall@1	Recall@3	Recall@5	Recall@10	MRR
easy	10	1.000	1.000	1.000	1.000	1.000
metadata	5	1.000	1.000	1.000	1.000	1.000
adversarial	6	1.000	1.000	1.000	1.000	1.000
cross-topic	8	0.583	0.625	0.813	0.958	1.000
ambiguous	5	0.467	0.583	0.683	0.950	0.900

---

🔴 Critical Finding: Duplicate Slot Waste

The single biggest issue this benchmark exposed:

Average wasted slots (same doc, diff chunk): 6.6 / 10
Max wasted slots: 8 / 10
Queries where >50% slots are duplicates: 25/34 (74%)

6.6 out of 10 result slots are occupied by different chunks from the same document. This is the primary driver of poor cross-topic and ambiguous recall — relevant docs from other documents never get a slot.

Deduped Recall (simulated maxChunksPerDocument=1)

If we deduplicate results by document (keeping only the top chunk per doc), recall improves dramatically:

Metric	Raw	Deduped	Delta
Recall@3	0.850	0.956	+0.106
Recall@5	0.909	0.983	+0.074
MRR	0.985	0.985	—

Recall@3 jumps from 0.85 to 0.96 just by deduping. This is the single highest-impact, lowest-effort improvement available.

---

Worst-Performing Queries

Query	Category	Recall@5	Issue
"best practices for API security"	ambiguous	0.25	Expected 4 docs (OWASP, JWT, rate limiting, OAuth). Got REST API first (not in expected set), then OAuth. JWT and OWASP buried by duplicates
"full-text search in databases"	cross-topic	0.33	Expected PG FTS + SQLite FTS5 + PG Indexing. PG FTS monopolizes top slots, SQLite FTS5 not surfaced
"CI/CD pipeline with automated testing"	cross-topic	0.50	Expected GH Actions + Integration Testing. GH Actions chunks fill most slots
"React performance and rendering"	ambiguous	0.50	Expected Perf Optimization + RSC. Perf doc's 5 chunks fill slots before RSC gets one

Pattern: Every low-recall query fails because a single highly-relevant document fills all slots with different chunks, preventing secondary docs from surfacing.

---

Analysis for the Improvement Plan

1. Improvement 0 (Benchmark): This benchmark is ready to be codified as a regression gate. The expanded corpus + query set covers the failure modes well.

2. maxChunksPerDocument default should be 2-3 (not unlimited): This is arguably Improvement 0.5 — a config change that would give the single biggest recall lift (+10.6% on Recall@3) with ~5 lines of code. Consider adding this as a quick win before tackling chunking changes.

3. Improvement 6 (Hybrid RRF): Would help cross-topic queries because FTS5 might rank different documents than vector search, increasing diversity after fusion.

4. Improvement 4 (Title Boosting): "best practices for API security" fails partly because "Web Application Security (OWASP)" has the word "security" in the title but gets no title boost. Title boosting would help here.

5. Easy/metadata/adversarial queries already at 100%: The current vector search is excellent for single-topic, direct-match queries. The improvements target cross-topic diversity.

6. Metadata queries passed at 100% because library filters are applied as SQL WHERE clauses (post-vector search), which works well. The metadata embedding improvement (Improvement 1) would help unfiltered metadata queries (e.g., "react hooks" without a library filter).

Raw Data

Key failure pattern in every struggling query — duplicate chunks monopolize results:

"React performance optimization" →
  Got (raw 10): [React Perf, React Perf, React Perf, React Perf, React Perf, React Perf, ...]
  Got (deduped): [React Perf, React Server Components, React useState Hook]
  // RSC is right there in position 2 if you dedup!