Skip to content

v0.4.0

tagged this 15 Aug 15:27
Idin's correction, and it was right: the goal was never to rank
results well. Precision is recoverable by whoever reads the results —
a useless result costs a moment — while recall is recoverable by
nobody, because nothing downstream can retrieve what was never
returned. A hundred results of which ninety are garbage is fine; ten
results missing the one that mattered is not.

Reciprocal rank fusion is gone. Fusing to a single ordering only
matters when returning a short list, and returning a short list was
the mistake. Three places were destroying recall, all of them cuts I
had chosen: a fuzzy floor at 0.72, a candidate pool capped at 20, and
a final limit of 5.

Selection is now a cascade. Every method runs, and each fills its own
quota from what earlier methods did not already claim — so a chunk
appears once, at the strongest method that found it, and no method
spends its quota on chunks another already returned:

  exact         uncapped — as good as a match gets
  starts_with   shortest chunks first
  ends_with     shortest chunks first
  contains      shortest chunks first
  fuzzy         by score
  cosine        by similarity

Anchored tiers order by chunk length because the query is most of a
short chunk and a fraction of a long one — length says which is the
better match without needing a model to judge. Idin's rule.

Semantic runs last, not because it matters least but because a chunk
whose words actually appear has already been found by a method that
can say so. What reaches that tier is what nothing else could find.

All tunable numbers move to search_config.ts. The candidate pool
sizes are computed from the quotas rather than stored, since a stored
copy drifts from the numbers it was meant to follow.

Also adds restWorkersAiEmbedder, so everything the Worker does can be
run from a terminal. A binding-only dependency made the embedding —
the number this whole design rests on — unverifiable outside the
runtime.

Verified against the real store with real embeddings: "canine" over
402 chunks returns 77 results, and all three frodo.md chunks are
found by cosine at 0.57-0.59 — by meaning, not by the "cannot"
letter-coincidence that had been retrieving them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Assets 2
Loading