Repository navigation
v1.4.0 — Statistical honesty + probe quality + integrity checks
Five improvements to measurement honesty, probe quality, and pack integrity.
Statistical honesty — confidence intervals on all pass rates. Pass rates reported as point estimates on n=10 samples are statistically meaningless (90% at n=10 has a 95% CI of 56%–100%). All rate thresholds now require a minimum sample of 20 and report a 95% Wilson score CI. coverage: 100 requires the CI lower bound ≥85% for spot-check, not just the point estimate ≥95%. index.json gains *_ci_low, *_ci_high, *_n fields for every metric.
High-stakes content bias in omission probe. Sampling was previously uniform random — wasting probes on boilerplate while missing the dates, amounts, and decisions that matter. Probe now samples in priority order: Tier 1 (dates, amounts, proper nouns, deadlines, decision statements) → Tier 2 (section headings + following paragraph) → Tier 3 (random uncited lines as fill). Minimum probe size raised to 20 slices. Tier 3 random misses only count as FAIL if substantively important.
Corpus-level negative space. Per-file Does-NOT-cover sections existed but were never aggregated. INDEX.md now has a "Corpus does NOT cover" section — derived from all per-file negative space entries, omission probe soft misses, and unresolved dangling references. Gives users explicit confirmation that a topic was looked for and not found, rather than leaving ambiguity between "absent" and "not checked."
Note integrity hashes. manifest.json hashed source files but not generated notes — a manually edited or corrupted note was invisible. Every per-file note is now SHA-256 hashed at write time. Step 6 re-hashes all notes and flags any mismatch as a Gap with LOW confidence demotion. Detects post-extraction edits, tool bugs, and disk corruption.
Ask mode miss logging. When ask mode falls back to a direct source read because notes were incomplete, the gap now gets logged to the relevant per-file note's Open questions with timestamp, question text, and citation. Builds a usage-driven record of what users ask that the pack failed to capture — picked up automatically by the next diff run.