Skip to content

v0.12.0 - The Stress Test

Latest

Choose a tag to compare

@eugeniughelbur eugeniughelbur released this 11 Jul 21:59
f4966a3

We stress-tested our own tool with 32 AI agents. Then fixed what they found.

A 4-lane adversarial audit (every command file, hostile synthetic vaults against every script, live retrieval quality on a 2,300-note vault, and a competitor teardown) produced 175 findings. This release closes ~160 of them across 24 PRs (#100-#123), in four phases.

Phase A - Data safety: tools that cannot eat your notes

  • heal_links/triage_links could permanently corrupt non-UTF-8 notes and silently rewrite line endings: strict UTF-8 in, byte-exact out, with non-UTF-8 files skipped loudly (#100)
  • One dangling symlink no longer kills the health check, the healers, or the exporter (#101)
  • The link healer only auto-fixes certain matches now: no more foreign-alphabet links "matching" unrelated notes, no [[C++]] -> C.md, no edits inside code fences (#102)
  • A leading BOM can't hide frontmatter; exports translate faithfully or say they failed; stats stopped double-counting after exports; link_graph actually mirrors vault_health (pinned by a drift-alarm test) (#103-#107)
  • A fresh bootstrap passes its own health check with zero findings - now a CI invariant (#108)

Phase B - Retrieval: search 2-4x better, measured honestly

  • The eval harness itself was lying (its "lexical" mode was secretly fused; "hybrid" double-counted semantic). Straightened first, so every claim below is real (#109)
  • Query-aware defaults: exact tokens go lexical, semantic votes lead the fusion (#110, re-tuned in #116)
  • The silent 2,000-file scan cap is gone; ranking learned that volume is not relevance (logs fade, dossiers speak per-section); the index reached 100% coverage via adaptive chunk-splitting (#111-#113)
  • Freshness: "current"-intent queries stop surfacing superseded notes and declined offers above the current truth (#114)
  • Multilingual default model (bge-m3): non-English queries went from ~0 to working, and English improved too. Migration: ollama pull bge-m3, then rebuild the index once (--build, ~10 min; existing indexes keep answering correctly until you do) (#115)
  • The reference numbers are committed in scripts/eval/BASELINE.md: keyword recall@10 1.0, paraphrase recall@10 0.77, non-English recall@5 0.63 (#116)

Phase C - Commands and docs tell one truth

  • 37 hardcoded folder paths across 18 commands now resolve via the folder map - /obsidian-export no longer produces an empty snapshot on default-bootstrapped vaults (#117)
  • 50 banned characters purged from instruction files - the skill stops teaching agents to write what its own validation hook blocks (#118)
  • Schema reconciliation: 4 missing note-type schemas added, 8 commands aligned, and the real-world exceptions (quick capture, kanban boards, raw sources, vault-surface files) are now documented law the hook understands (#119)
  • 20 "manual for a different machine" bugs fixed, including a status check that read its own output as evidence (#120)
  • SKILL.md's six drifted sections now defer to the command files as the single source of truth; README counts match the filesystem; builds ship no Python bytecode (#121)
  • Five new CI lints enforce all of the above permanently

Phase D - The front door

  • The README's one-line installer pointed at a file that did not exist. It exists now (idempotent), prerequisites are stated up front, there's a path for users with no vault yet, and the bootstrapper never overwrites your files - keep-by-default, --force is the consent (#122)
  • Every platform dist ships a runnable Python project, so the research toolkit works on Codex, Gemini, OpenCode, Hermes, and Pi - not just Claude Code (#123)

After the sprint closed (same day)

  • External contribution (one of 17 outside contributors to date): /research and /research-deep silently ignored a PERPLEXITY_API_KEY set in the config .env - the free-vs-paid decision ran before anything had loaded the file. Root-caused and fixed by @MichaelHabermas (#125, fixes #124), now fenced in CI
  • The audit's three never-tested surfaces (bg-agent hook, MCP write path, Telegram ingest) ran live; the one real bug found - Telegram ingest hardcoding wiki-style folders - is fixed with folder-map resolution (#126)

Numbers

  • Test suite: 30 -> 129 tests, including 11 permanent invariants (byte-safety, ghost resilience, drift alarms, folder/character/schema/roster lints, the showroom rule, front-door safety, dist runnability)
  • Full changelog: see CHANGELOG.md under 0.12.0

Breaking / migration notes: the default embedding model is now bge-m3 (OBSIDIAN_EMBED_MODEL overrides; one-time index rebuild recommended); vault_stats accepts --path (old --vault still works); bootstrap keeps existing files by default (use --force for the old overwrite behavior); the semantic index format is v2 (rebuilds automatically invalidate old caches).