Search Measurements

Search retrieval measurements

What retrieval scores, what has been tested against it, and what a probe has to do to be believed. The scorecard is the verdict metric for extraction quality, not the cheap heuristics — see Extraction-Pipeline for why those cannot see the dominant defect. Query mechanics: Search-Query-Construction.

The harness

src/scripts/score-retrieval.js, run on the devbox VM (Operating-DEMI-Data-Plane § Reach the data plane) because the data plane is private and keyless. Label sets are committed as src/scripts/retrieval-labels-{A-text,B-ocr-legacy,C-ocr-pdfium,D-ocr-tiled,E-control-textless}.jsonl, so every number here is reproducible from a checkout.

Labels come from an independent reader rather than from a human. The requirement is narrower than "a human wrote it": the phrase must be independent of the pipeline under measurement, because a phrase lifted from docling's own markdown retrieves itself by construction. Two readers satisfy that and share no code with docling — poppler's pdftotext where a text layer exists, and a pdftoppm page render read by eye where it does not.

Strata are what the extraction sidecars can prove. pdf_backend is recorded on 2,755 of 51,460 sidecars (measured 2026-08-04), and the field started being written seven hours after the backend switch, so a timestamp alone does not separate the backends. That seven-hour window is excluded from the strata rather than guessed at.

E-control-textless is a negative control: three documents extracted to nothing but image placeholders. It scores 0 in every arm of every experiment, which is what makes the rest meaningful — without it, a uniform ~0.5 could not be told apart from a broken harness.

Scorecard

Pooled across the labelled strata, control excluded, n=78, measured 2026-08-05:

	recall@1	recall@5	recall@10	recall@20	recall@50	MRR
BM25 alone	0.308	0.513	0.590	0.641	0.705	0.398
BM25 + semantic ranker	0.372	0.603	0.628	0.680	0.705	0.472

One standard error is ~0.056 at n=78, so each individual gain is inside noise. The case for the ranker is the consistent direction across all three headline metrics with nothing regressing: 5 miss→hit against 2 hit→miss at k=10, 23 labels up and 7 down, top-1 hits 24 → 29.

found@50 is identical at 55 in both arms, and that is the load-bearing check: a reranker can only reorder what BM25 already retrieved, so an unchanged found-count is the evidence that L1 was untouched. Had it moved, the query layer would have changed underneath the comparison and the two arms would not be comparable.

Every label is a verbatim phrase lifted off a page — exact-match lexical retrieval, the weakest case for a ranker trained on natural language — which makes the measured gain a floor rather than an estimate. What real users type is untested, and there is no query log to build that label set from.

How the semantic ranker is wired

• It uses semanticQuery, not queryType: 'semantic'. The latter accepts only plain text and rejects both simple and full Lucene syntax, so it would silently discard the fuzzy arm, the boost, the stopword guard and search-as-you-type — every measured retrieval fix on this page. semanticQuery leaves the L1 query untouched and reranks separately. Its value is the tokenized terms rejoined, not the raw keywords, because operator syntax inside the semantic string is unsupported.
• It is skipped on matchAll searches: search: '*' has no relevance signal to rescore, and Azure bills per non-empty semantic query.
• It degrades without failing. On throttling or model error the service returns the BM25 order with @search.semanticPartialResponseReason and HTTP 200; Basic tier allows 2 concurrent semantic requests per search unit. HTTP 402 means the monthly allowance is spent, and a UTC-month latch stops asking for the rest of that month. GET /db/stats reports the counters under search.semantic — requested, partial, ranked, exhausted, and the last degrade reason.

Where the misses go

Four explanations were tested. Three were rejected, and the order matters because each rejection narrowed the next.

Index coverage — ruled out. searchChunks({matchAll: true}) reports 1,128,736 rows in the chunks index against 1,128,733 chunks in Cosmos (measured 2026-08-23). The PT5M indexer keeps up.

The strict conjunction — ruled out. The hypothesis was that joining terms with AND caps recall before BM25 ranks anything, which is structurally true and predicts a ~40% loss independent of extraction path. Tested by flipping the join to OR, both arms paired in one session against one index state, measured 2026-08-04: pooled recall@10 moved 0.549 → 0.577, half a standard error, on 4 miss→hit against 2 hit→miss. The discriminating text stratum did not move at all. recall@1 and MRR both got worse — OR admits candidates, it does not place them.

Four guards held. The AND arm reproduced the baseline exactly, the negative control stayed at 0 in both arms, and median matchingChunks per label went from ~15–21 under AND to 525,000–775,000 under OR, which proves the knob reached the wire. A mechanism check refutes the hypothesis independently: bucketing labels by whether they contain a token tokenize splits on punctuation (5.1, mg/L, PH12-3-3) puts the entire nominal lift in the wrong bucket.

The result is stronger than closing one hypothesis. Under the OR arm the candidate set is effectively the whole corpus and BM25 alone chooses the top 10, and the missed documents stayed missed. A document excluded by a filter reappears when the filter is removed; these did not. There is no anyTerms knob in the code — a live search parameter that exists only to be false is maintenance, not a finding.

The phrase not being in the chunk — ruled out. src/scripts/probe-phrase-presence.js reads each labelled document's chunks from Cosmos and finds the strictest rung at which the phrase is present: exact → whitespace → punct → despaced. Measured 2026-08-04:

	phrase present	despaced	straddle-*	not contiguous
retrieval hit (n=39)	32	2	0	5
retrieval miss (n=32)	25	3	0	4

78% of the misses have the phrase sitting verbatim in a stored chunk, and on the text strata it is 16 of 17. Extraction is not what costs recall here. Word-joining accounts for 3 of 32 misses — real, measured, and an order of magnitude smaller than its 23–29× token-rate figure implies for retrieval.

straddle-* is zero: not one label in 74 was present only across a chunk seam. Chunk overlap is a real chunker concern and costs approximately nothing in retrieval. The ~3–4% figure sometimes quoted for it was never measured; the measurement is 0/74.

The fuzzy term against an analyzed index — confirmed. buildQuery emits (term OR term~1), and a fuzzy term bypasses the query analyzer, so ~1 demands a literal against an index side that is lemmatised and stopword-stripped under en.microsoft. For a term the analyzer removes, the analyzed half of the clause contributes nothing, which leaves the unanalyzed ~1 half as the only thing the clause can match — and it matches by edit distance against whatever unrelated tokens sit within distance 1. The clause becomes a near-random mandatory filter.

Confirmed on one document. Label "Sediments from the proposed Lodgepole mine will move downstream and accumulate", one chunk, holding that sentence verbatim and indexed. Querying the full phrase restricted to that document returns 0 with fuzzy: true and 1 with fuzzy: false. from matches 81,731 chunks corpus-wide but 0 inside the target document, so the conjunction discards the one right answer.

Two things that look like the cause and are not, both measured:

• Short stopwords are innocent. the, of and and return 0 as single-term queries, which looks damning, but adding them to a real query changes its hit count by nothing (9132 → 9132, 126 → 126). They fall under MIN_FUZZY_LENGTH so they never get a variant, and the analyzer drops them. A single-term probe cannot tell "analyzer removed it" from "matches nothing".
• Index coverage, again. All three zero-match documents had every Cosmos chunk present in the index.

ANALYZER_STOPWORDS is measured, not guessed: 20 terms at or above MIN_FUZZY_LENGTH. Sweeping all 360 distinct ≥4-char terms in the label corpus finds exactly 6 analyzer-removed words, 5 of them already listed. The sixth is monirose, OCR garbage that returns 0 because it is genuinely absent rather than because it is a stopword — and a rare misspelling is exactly the case fuzzy exists to rescue, so it stays out of the list.

Why the fuzzy arm is down-weighted

Blanket --no-fuzzy scores 2 labels higher than the stopword guard alone (44 against 42) while the vocabulary sweep comes back clean, so the residue is not more stopwords. The mechanism is fuzzy diluting BM25 on ordinary terms — a ranking effect rather than a zeroing one: if the fuzzy arm competes with the exact arm on score, a document matching only by edit distance can outrank one holding the term verbatim. FUZZY_BOOST keeps ~1 but scores it lower, (term OR term~1^0.5), and lands on the same 44 labels blanket --no-fuzzy reaches without giving up typo tolerance.

Measured 2026-08-04, n=71: pooled recall@10 0.592 → 0.620, recall@1 0.282 → 0.310, MRR 0.382 → 0.403, on 2 miss→hit and 0 hit→miss, no stratum regressed. Eight ranks moved: five improved, two slipped slightly but stayed on the page (9→10, 7→8), and one miss came back at 10. The two slips are the mechanism working as intended — fuzzy-only matches demoted beneath exact ones.

FUZZY_BOOST is a constant, not a request parameter. The bar it cleared, stated before the run, is the bar any retrieval change has to clear: recall@10 improves and neither recall@1 nor MRR regresses.

Known limits

The text stratum loses about 40% of labels at top 10, measured 2026-08-04 at n=39 with one SE ~8 points, so it is a measurement rather than a small-sample artefact. Word-joining does not explain it, since those documents barely join. The cause is unknown. Two explanations are tested and rejected: that misses cluster on structured page furniture rather than running prose (0.50 at n=10 against 0.62 at n=29, well inside noise), and that label length predicts it (0.75 / 0.43 / 0.53 by word count at n=46 — noise).

There is no labelled OCR-scan stratum. Building one needs page renders of scans, and a large share of the scanned source PDFs return 404 in the object store. Titles cannot substitute: they are metadata, never verified to be on the page, so scoring them would measure the title.

Tiled retrieval is binary, and one miss is a pure ranking failure. recall@1, @5 and @10 are all 0.333 on the tiled stratum (n=9, measured 2026-08-04): every hit ranks first, nothing lands at 2–10. Cross-tabulating probe-phrase-presence.js against the scorecard splits the 6 misses into 4 where the text really is absent from the extraction, 1 word-joining, and 1 where the phrase is present verbatim — 58869332de49fe015163a0c9, class: exact, coverage 1.0, a single chunk, not truncated — and the document still does not reach the top 10. Two of the 3 hits go the other way, surfacing without the phrase being fully extracted. On this stratum, presence in the extraction and presence in the results are close to independent, so more extraction effort will not fix the ranking half.

Rules a probe has to follow

Each rule comes from a probe that produced a confident wrong answer.

1. A probe harness that reuses one output file lies. A timed-out curl does not rewrite its -o file, so every probe re-reads the previous probe's JSON. That produces a table of identical hit counts, which reads as "the results are wrong" when the truth is "the app is hung". One output file per probe, always, and abort the whole run on the first unhealthy probe.
2. Instrument before deploying on a hypothesis. Four fixes shipped on plausible causes and all four failed. Both turning points were adding a measurement — indexProgress, then fuzzyEnabled.
3. A probe that counts rows cannot see a malformed row. Read rendered output, not just counts.
4. A probe that cannot fail proves nothing. Four instances:
   • keywords=<exact term>&fuzzy=true returns 1 whether fuzzy is on or off. The probe that settled it was a nonsense token one edit from an indexed one, unreachable by stemming.
   • A probe returning the top cap proves nothing. An ineffective full-text predicate returns the whole container ranked, which is why the Cosmos spike concluded fuzzy worked when it did not.
   • A latency claim needs a before reading. "/db/stats answers in seconds, not minutes" was offered as proof Phase 8 landed; the legacy build answered in 2.2 s too. The payload discriminates, not the timing.
   • A capped list hides an ACL difference. Comparing anonymous and privileged result counts proves nothing when the page caps at 10 and every row is public. Use a throwaway record with a non-public read[] and assert 0 against 1.
5. Check what a probe can distinguish before reading its answer. The self-phrase probe proved the missed documents are indexed, and was then read as proving the cause is word-joining, which it cannot: a self-phrase is contiguous within one chunk and composed of tokens that survived extraction by construction, so every competing hypothesis predicts it succeeds equally.