v2026.05.20.3

What's in this build

Built from 40233aada84298e94a232e5956e9c9c8cd4097bb on 2026-05-20T23:51:46Z.

Changes since v0.21.1

• fmt: cargo fmt across workspace post-v0.21.1

v2026.05.20.2

Fathom v0.21.1 — release notes

Tag: v0.21.1
Date: 2026-05-20
Branch: merged from feat/v0.21.1-lexicon-and-bm25 (19 commits)

TL;DR

Search is significantly better. Iconic philosophical quotations now reliably surface their source book at rank 1. Recall and precision both improve materially across the bench. Shard format unchanged — no re-download, no rebuild required.

What's new

Hybrid retrieval — BM25 + dense, fused via RRF

The search bar now runs two retrieval lanes in parallel and fuses the results:

• Dense lane — existing bge-small-en-v1.5 cosine similarity over chunk embeddings.
• BM25 lane — new, runs per-shard against a custom tokeniser that preserves stopwords + bigrams. Built eagerly when a shard is loaded, cached for the shard's lifetime in the LRU.
• Fusion — reciprocal rank fusion (RRF) with k=30 by default, picked after a 14-configuration calibration sweep. Linear convex combination available as an opt-in via env vars.

Fusion mode and parameters configurable at runtime via FATHOM_FUSION_MODE, FATHOM_FUSION_ALPHA, FATHOM_RRF_K.

Iconic-phrase alias dictionary

Five known canonical quotations now hard-pin their source book at rank 1, bypassing retrieval:

• "the unexamined life is not worth living" → Plato, Apology
• "I think therefore I am" / "cogito ergo sum" → Descartes, Discourse / Meditations
• "the only thing I know is that I know nothing" → Plato, Apology
• "appearances are not realities" → Plato dialogues

Operator-curated, in-binary TOML. Easy to grow as canon-class queries surface in real use.

Bench harness extended

fathom-bench gains --fusion-mode, --fusion-alpha, --rrf-k flags for sweeping. Calibration result for v0.21.1 committed at crates/fathom-bench/results/v0.21.1-final.json with decision notes at v0.21.1-final.notes.md.

SelectionPopover hardening (bundled)

Two desktop UX fixes hitched a ride on this release:

• .page-window overflow clip — eliminates ghost text in the column gutter on wide displays.
• SelectionPopover max-height now derives from the chosen top so the popover never extends past the viewport on selections low on the page.

Independent of the BM25 work; bundled because both were ready at branch time.

Bench results

Measured against the live R2 corpus (551 books, ~263k chunks) using crates/fathom-bench/data/queries.json (20 iconic-phrase queries, 10 topical, 5 adversarial).

Metric	v0.20-baseline	v0.21.1	Delta
recall@50	76.5%	88.2%	+12pp
MRR	0.348	0.533	+53%
hits@1	29.4%	47.1%	+18pp
hits@5	35.3%	58.8%	+23pp
hits@10	52.9%	76.5%	+24pp
adversarial max similarity	0.678	0.032	20× cleaner noise floor
iconic queries surfaced top-10	13/17	13/15	flat (see below)

Some of the headline gain is the dense lane having improved between v0.20 and v0.21.1 (bge-small re-embedding pipeline, not the BM25 work specifically). Pure-dense recall@50 is 94.1% — fusion costs 6pp of top-50 recall in exchange for tighter top-10 ranking and a much lower adversarial noise floor.

Known tradeoff: one query regresses

The query "what is past is done with" (Marcus Aurelius, Meditations VIII.36) ranks #42 in dense-only mode and falls out of top-50 entirely under any fusion mode. Aristotle's Nicomachean Ethics contains the verbatim phrase "what is done and past" and BM25 promotes Aristotle above Marcus. This is a real lexical-vs-semantic conflict; no fusion mode in the sweep fixes it. Worth knowing if it bites you — workaround is FATHOM_FUSION_MODE=dense_only.

Known issues (deferred to v0.21.2)

Surfaced during UAT, specs filed under docs/superpowers/specs/:

• Simple-tier paraphrase context bleed — Simple tier sometimes pads short selections by drawing from adjacent chunk context, and can shift register away from doctrinal claims (5/05/20-simple-tier-paraphrase-bugs.md).
• Selection boundary leak — when UAX#29 sentence-snap fails, the fallback to raw byte clamp can produce mid-sentence cuts in the paraphrase input. Same spec.
• Catalogue search gap — searching by book title or author name doesn't reliably surface that book; book-level metadata never enters retrieval. Spec at 2026-05-20-catalogue-search-guard.md.
• alias_hits excerpt quality — iconic-guarded results show the book's front-matter chunk as the excerpt, not the chunk where the phrase actually appears. Cosmetic but visible.

None block v0.21.1. All scoped for v0.21.2.

What didn't ship from the original spec

The v0.21.1 design originally bundled a lexicon expansion campaign (~150 books with substrate + doctrinal-source + quote schema, MARCH-pattern verifier pipeline). Deferred to v0.22 so the BM25 slice could ship cleanly. Lexicon spec and plan remain in docs/superpowers/specs/ and docs/superpowers/plans/ for fresh-session pickup.

Migration

• No shard format change. Existing v0.2 corpus continues to work. No re-download.
• No lexicon change. The 30 pre-v0.21.1 lexicon files still ship as-is; v0.22 will migrate them to the expanded schema.
• New env vars are optional. Defaults match the calibration winner; unset env vars produce identical behaviour.

Tests

• 134 Rust tests across the workspace, all passing
• 34 Svelte vitest tests, all passing
• svelte-check 0 errors, 0 warnings

Architecture notes

For future contributors:

• BM25 is per-shard, built lazily via tokio::task::spawn_blocking inside ensure_shard. IDF is local to each shard, not corpus-wide — fine for iconic-phrase recall, slightly distorts cross-book topical queries.
• The custom FathomTokenizer in crates/fathom-core/src/bm25_index.rs bypasses bm25 crate's default English stemmer + stopword filter entirely. Stopwords are load-bearing in philosophy (Heidegger's "Being and Time" fails any stopword filter).
• Fusion lives in crates/fathom-core/src/fusion.rs. RRF is the default; linear CC is available but parameter-sensitive.
• Shard was split into ShardWire (Deserialize, no bm25 field) and runtime Shard (with Arc<ShardBm25> field). Wire compatibility unchanged.

Acknowledgements

Calibration sweep run on the live R2 corpus 2026-05-20. Spec + plan written same-session (docs/superpowers/specs/2026-05-20-fathom-v0.21.1-lexicon-and-bm25-design.md + matching plan).

v2026.05.20.1

What's in this build

Built from 8a850111e446bb0bd70dfb215eee2b5d3f1d100b on 2026-05-20T00:41:21Z.

Changes since v2026.05.19.2

• desktop: harden selection popover + cap reading measure + broaden exit-event hook

v2026.05.19.2

What's in this build

Built from c88e9809552dc3d4ac11b6b73dad1732f0c9ec4b on 2026-05-19T22:58:24Z.

Changes since v2026.05.19.1

• desktop: floating SelectionPopover + CSS-column pagination + client sentence-snap
• docs: refresh README for v0.21

v2026.05.19.1

What's in this build

Built from a86d3ac322b4ed75a1f52100126959a391646497 on 2026-05-19T01:23:21Z.

Changes (initial dated release, last 20 commits)

• ci: date-based v* tags on every push to main + commit-list changelog
• desktop: per-model ready flags + modal waits for explicit Get started
• desktop: per-component download bars + low-RAM serialisation across embedder+warmup
• desktop: warm Gemma + DeBERTa at boot so onboarding gates the real download
• desktop: search bar stops narrating embedder state
• desktop: use-system-status init() guard against double-init
• desktop: StatusDot + use-system-status
• desktop: Onboarding spinner respects prefers-reduced-motion
• desktop: Onboarding modal mounted on first launch
• desktop: use-onboarding catches invoke errors in complete()
• desktop: onboarding sentinel commands + use-onboarding store
• desktop: LeftRail focus-visible outlines on book/hit/retry buttons
• desktop: LeftRail replaces LibrarySidebar with theme picker
• desktop: use-themes cache test now exercises the cache path
• desktop: use-themes class-instance store
• classify-themes: full 551-book classification
• desktop: runtime stripper for Gutenberg preamble + license footers
• core+engine: anti-fabrication guardrails for Simple paraphrase
• classify-themes: 50-book validation pass
• desktop: tier-detection tests, env override, startup logging, cold_load helper

Latest build from main

What's in this build

Built from 40233aada84298e94a232e5956e9c9c8cd4097bb on 2026-05-20T23:51:46Z.

Changes since v0.21.1

• fmt: cargo fmt across workspace post-v0.21.1