The overview learned to ignore noise
napkin overview is the agent's vocabulary primer: one line per folder, with the TF-IDF keywords that make that folder distinctive, so the agent's next lexical search is phrased in the vault's own words. On real-world vaults full of imported documents — OCR'd PDFs, e-signature exports, HTML conversions — that primer used to drown: dashed GUIDs shattered into short "words" that won TF-IDF, and hundreds of near-identical converted-document folders buried the curated structure.
v0.9.0 fixes this with statistics, not blacklists.
✨ Structured-noise stripping
GUID/dashed-hex runs, digit-mixed ID blobs, HTML tags & entities, and long hex-letter runs are removed from text before tokenization, so ID shrapnel never reaches keyword scoring. Real short vocabulary (pdf, xlsx, fee) is untouched — noise is recognized by its structure in context, never by judging tokens.
✨ Homogeneous-sibling collapse
When a folder has many children whose content (body + headings, never filenames) is statistically near-identical, they render as one honest row:
imports/contracts/ (+153 similar subfolders)
keywords: agreement, signature, lease, addendum
The vault root is never collapsed — top-level folders are your taxonomy. Configurable via overview.collapse (default true) or --no-collapse; JSON output gains an additive collapsedFolders field.
🐛 --json is now always parseable
Warnings (e.g. malformed YAML frontmatter) go to stderr instead of corrupting stdout.
Measured on real agent vaults
| worst-polluted vault | clean vaults | |
|---|---|---|
| overview lines | 242 → 11 | unchanged |
| noise keywords | 778 → 6 | 0 |
| curated-keyword robustness | 0.71 → 0.82 | 1.0 |
If your vault is clean, this release changes nothing. If it isn't, your agents just got a much better map.