Skip to content

v0.8.1

Latest

Choose a tag to compare

@github-actions github-actions released this 04 Aug 21:39

NLI model swap — contradictions stop firing at strangers

  • The default NLI model is now deberta-v3-small-tasksource-nli
    (multi-task DeBERTa-v3-small, our quantized ONNX export, 172 MB),
    replacing mobilebert-uncased-mnli. MNLI-only models presuppose that two
    sentences co-refer, so unrelated notes in the same register scored as
    confident contradictions ("Engram Alpha is written in Rust" vs "TepinDB is
    published on crates.io": c=0.99, straight past the gate). Measured on the
    eval/CONTRADICTIONS.md harness, three seeds at the reference shape:
    false alarms at the shipped 0.80 gate drop 38% → 18–29% with catch at
    94–96% (100% ungated), agree-alarms 6–13% → 0–1% ungated, and on this
    repo's real graph the unbiased queue noise goes 28% → 0%. MobileBERT
    and the 0.7.2 DeBERTa stay selectable in the pane; explicit selections
    keep resolving.

This release's research cycle asked two questions at 100–2000 notes — can focus
survive scale, and can a false-positive rate of 1.00 be brought down without
cutting a single result — and this release ships the three answers the
tricks bench measured (eval/README.md has the full tables).

Transplant probes — the register gap closed at scale

  • The weak line now calibrates in the graph's own voice, and honest FP
    lands at 0.02 at 1500 notes (0.00 at 100), from 0.84/0.32.
    The
    weak-line fit mints two probe families and takes the max of their
    quantiles: the existing question templates over borrowed vocabulary, and
    transplants — real sentences from real notes with their two most
    distinctive words swapped for coinages (Inverse Cloze Task inverted,
    ACL 2019). Transplants score exactly like the loudest real noise
    register, so the line finally clears the crowd ceiling that templated
    probes under-read. Price, stated plainly: the "likely not in memory"
    hedge now leads ~45% of lowest-confidence answerable replies (answers
    still delivered, recall/focus unchanged); policy.weak_line_quantile
    is the per-graph softness knob.
  • Auto-tune is damped and glitch-bounded. Every dial now travels half
    the distance from its current value to the fresh fit per pass
    (policy::AUTO_TUNE_DAMPING = 0.5) and the damped target is
    hard-clamped into the dial's band — one noisy fit can no longer teleport
    a threshold, a corrupt value recovers on the next pass, and consistent
    fits still converge in a few session boundaries. The journal shows both:
    weak line 0.850 -> 0.663 (fit 0.476, damped).
  • The negative results are receipts too (eval/results/,
    --qpp mode is new): every per-query unanswerability signal measured
    blind at 1500 — score-shape features, pool-bottom z, random-background
    z/Gumbel, embedding coherence, local-crowd shoulder (AUC 0.60–0.68) —
    the crowd fakes the curve's shape, not just its scale. The knee buffer
    and full-note reranker input also died measured deaths. The eval README
    gained the evolution table: every generation, what changed, what it
    bought, refuted branches included.

Knee-mode delivery trim

  • Delivery now cuts at the score cliff, not just a fixed depth. After
    the fixed floor, the delivered list is trimmed at the largest relative
    drop in its score curve when that drop is at least policy.knee_cliff
    (default 0.25; null opts out) — a simplified Tail-Aware Adaptive-k
    (arXiv:2606.11907). Measured recall-free at every size from 100 to 2000
    notes (recall@5 and oblique recall within 0.01) while focus rises 3–4.5×
    and delivered tokens fall 35–50% — and unlike the fixed floor, the cliff
    sharpens as the graph grows, so the gain holds at scale.

The calibrated "likely not in memory" verdict

  • A weak search verdict now says what it means: this likely isn't in
    memory.
    The reply leads with that recommendation while the nearest
    candidates are still delivered, never cut — a label, not a barrier. On the
    bench this turns the honest false-positive rate on never-written questions
    from 1.00 into 0.08–0.12 at every measured size, at the price of a
    verify-first note on the lowest-confidence quarter-to-half of real
    questions.
  • The skills (all three variants + plugin copy) and the search tool teach
    the new verdict wording.

Auto-tune's second dial: the weak line

  • The weak-evidence line is now calibrated per graph. The measured
    correct line runs 0.56 → 0.81 from 100 to 2000 notes, so the fixed 0.85
    default was only ever right for big graphs. At session boundaries (past 50
    notes, reranker loaded) auto-tune mints deterministic phantom probes —
    questions about coined subjects that cannot exist in any graph — and fits
    policy.weak_evidence_top to the policy.weak_line_quantile (default
    q90) of what they still score, split-conformal style. One
    policy.auto_tune button governs both dials; every move lands in one
    journaled auto_tuned row.

Pane

  • Graph settings → Calibrated delivery grew the knee-trim toggle with its
    cliff stepper and the weak-line quantile, with the plain-word explanations
    rendered from the live values (carried from earlier in this cycle: the
    calibrated-delivery settings block itself).

Measured end to end, and the field lesson

  • New --posttune eval mode measures the shipped stack — knee on, weak
    line auto-calibrated, FP under the recommendation regime — as one
    engram-only pass per size; the eval README's baseline tables now carry
    engram (pre-tune) and engram (post-tune) rows at 100 and 1500
    notes. Post-tune at 1500: 315 tok/query (−40%), focus 0.10→0.44, recall
    −0.01. At 100: 192 tok/query, focus 0.20→0.64, honest FP 0.96→0.32.
  • Phantom probes speak the graph's own language. The first live fit
    showed cross-encoder score scales are register-dependent (a real graph's
    noise ceiling sat at ~0.20 where the eval corpus's sat at 0.77), so the
    weak-line probes now borrow vocabulary from the graph's own note titles —
    the coined subject keeps them unanswerable, the borrowed words keep them
    in register — and the fitted line clamps floor-relative
    (delivery_floor + 0.03), never to an absolute. Known open edge: at 1500
    notes the shipped calibration still under-reads the crowd ceiling
    (end-to-end FP 0.84 vs the 0.12 in-register ceiling) — named as the next
    cycle's problem in the eval README.
  • PLAN.md retired: merged into a slim CLAUDE.md keyword index — the memory
    graph is the plan of record (PLAN §-references in code comments point at
    git history).

Full Changelog: v0.8.0...v0.8.1