NLI model swap — contradictions stop firing at strangers
- The default NLI model is now
deberta-v3-small-tasksource-nli
(multi-task DeBERTa-v3-small, our quantized ONNX export, 172 MB),
replacingmobilebert-uncased-mnli. MNLI-only models presuppose that two
sentences co-refer, so unrelated notes in the same register scored as
confident contradictions ("Engram Alpha is written in Rust" vs "TepinDB is
published on crates.io": c=0.99, straight past the gate). Measured on the
eval/CONTRADICTIONS.mdharness, three seeds at the reference shape:
false alarms at the shipped 0.80 gate drop 38% → 18–29% with catch at
94–96% (100% ungated), agree-alarms 6–13% → 0–1% ungated, and on this
repo's real graph the unbiased queue noise goes 28% → 0%. MobileBERT
and the 0.7.2 DeBERTa stay selectable in the pane; explicit selections
keep resolving.
This release's research cycle asked two questions at 100–2000 notes — can focus
survive scale, and can a false-positive rate of 1.00 be brought down without
cutting a single result — and this release ships the three answers the
tricks bench measured (eval/README.md has the full tables).
Transplant probes — the register gap closed at scale
- The weak line now calibrates in the graph's own voice, and honest FP
lands at 0.02 at 1500 notes (0.00 at 100), from 0.84/0.32. The
weak-line fit mints two probe families and takes the max of their
quantiles: the existing question templates over borrowed vocabulary, and
transplants — real sentences from real notes with their two most
distinctive words swapped for coinages (Inverse Cloze Task inverted,
ACL 2019). Transplants score exactly like the loudest real noise
register, so the line finally clears the crowd ceiling that templated
probes under-read. Price, stated plainly: the "likely not in memory"
hedge now leads ~45% of lowest-confidence answerable replies (answers
still delivered, recall/focus unchanged);policy.weak_line_quantile
is the per-graph softness knob. - Auto-tune is damped and glitch-bounded. Every dial now travels half
the distance from its current value to the fresh fit per pass
(policy::AUTO_TUNE_DAMPING = 0.5) and the damped target is
hard-clamped into the dial's band — one noisy fit can no longer teleport
a threshold, a corrupt value recovers on the next pass, and consistent
fits still converge in a few session boundaries. The journal shows both:
weak line 0.850 -> 0.663 (fit 0.476, damped). - The negative results are receipts too (
eval/results/,
--qppmode is new): every per-query unanswerability signal measured
blind at 1500 — score-shape features, pool-bottom z, random-background
z/Gumbel, embedding coherence, local-crowd shoulder (AUC 0.60–0.68) —
the crowd fakes the curve's shape, not just its scale. The knee buffer
and full-note reranker input also died measured deaths. The eval README
gained the evolution table: every generation, what changed, what it
bought, refuted branches included.
Knee-mode delivery trim
- Delivery now cuts at the score cliff, not just a fixed depth. After
the fixed floor, the delivered list is trimmed at the largest relative
drop in its score curve when that drop is at leastpolicy.knee_cliff
(default 0.25;nullopts out) — a simplified Tail-Aware Adaptive-k
(arXiv:2606.11907). Measured recall-free at every size from 100 to 2000
notes (recall@5 and oblique recall within 0.01) while focus rises 3–4.5×
and delivered tokens fall 35–50% — and unlike the fixed floor, the cliff
sharpens as the graph grows, so the gain holds at scale.
The calibrated "likely not in memory" verdict
- A
weaksearch verdict now says what it means: this likely isn't in
memory. The reply leads with that recommendation while the nearest
candidates are still delivered, never cut — a label, not a barrier. On the
bench this turns the honest false-positive rate on never-written questions
from 1.00 into 0.08–0.12 at every measured size, at the price of a
verify-first note on the lowest-confidence quarter-to-half of real
questions. - The skills (all three variants + plugin copy) and the search tool teach
the new verdict wording.
Auto-tune's second dial: the weak line
- The weak-evidence line is now calibrated per graph. The measured
correct line runs 0.56 → 0.81 from 100 to 2000 notes, so the fixed 0.85
default was only ever right for big graphs. At session boundaries (past 50
notes, reranker loaded) auto-tune mints deterministic phantom probes —
questions about coined subjects that cannot exist in any graph — and fits
policy.weak_evidence_topto thepolicy.weak_line_quantile(default
q90) of what they still score, split-conformal style. One
policy.auto_tunebutton governs both dials; every move lands in one
journaledauto_tunedrow.
Pane
- Graph settings → Calibrated delivery grew the knee-trim toggle with its
cliff stepper and the weak-line quantile, with the plain-word explanations
rendered from the live values (carried from earlier in this cycle: the
calibrated-delivery settings block itself).
Measured end to end, and the field lesson
- New
--posttuneeval mode measures the shipped stack — knee on, weak
line auto-calibrated, FP under the recommendation regime — as one
engram-only pass per size; the eval README's baseline tables now carry
engram (pre-tune) and engram (post-tune) rows at 100 and 1500
notes. Post-tune at 1500: 315 tok/query (−40%), focus 0.10→0.44, recall
−0.01. At 100: 192 tok/query, focus 0.20→0.64, honest FP 0.96→0.32. - Phantom probes speak the graph's own language. The first live fit
showed cross-encoder score scales are register-dependent (a real graph's
noise ceiling sat at ~0.20 where the eval corpus's sat at 0.77), so the
weak-line probes now borrow vocabulary from the graph's own note titles —
the coined subject keeps them unanswerable, the borrowed words keep them
in register — and the fitted line clamps floor-relative
(delivery_floor+ 0.03), never to an absolute. Known open edge: at 1500
notes the shipped calibration still under-reads the crowd ceiling
(end-to-end FP 0.84 vs the 0.12 in-register ceiling) — named as the next
cycle's problem in the eval README. - PLAN.md retired: merged into a slim CLAUDE.md keyword index — the memory
graph is the plan of record (PLAN §-references in code comments point at
git history).
Full Changelog: v0.8.0...v0.8.1