Repository navigation
🎯 Data Quality Fixes + Documentation Refresh
This release fixes data labeling issues discovered after v5.0.0 and brings all documentation up to date with the v5 architecture.
🔧 Data Fixes
- Relabeled 199 trustworthy-with-gap cases to abstain — cases where context had insufficient evidence were incorrectly labeled as trustworthy
- Removed 10 mislabeled single-context dispute cases — disputes require multiple sources; single-context cases cannot be genuine disputes
- Fixed 51 broken queries across the dataset
- Updated test suite to match new labels
📚 Documentation Refresh
- README.md: Added Mermaid evaluation flow diagram, emoji section headers, links to evaluation guide and mode decision tree
- docs/evaluation-guide.md: Full rewrite for v5 — 4 categories, cross-cutting quality checks, updated counts/difficulty/FAQ
- docs/mode-decision-tree.md: Updated category names (qualification → trustworthy_hedged, confidence → trustworthy_direct), rewrote grounding/relevance as cross-cutting checks
- docs/GOVERNANCE_CASE_TAXONOMY.md: Updated version, case counts, removed stale path and broken link
- CHANGELOG.md: Restored emoji consistency on v4.0–v5.0 subsection headers
- Archived completed v5-plan docs and multi-source proposal to
docs/roadmap/archive/ - Fixed stale difficulty breakdown (trustworthy_hedged: 435/725 → 428/732 medium/hard)
📊 Benchmark Totals (unchanged)
| Category | Cases |
|---|---|
| Abstention | 685 |
| Dispute | 675 |
| Trustworthy Hedged | 1,160 |
| Trustworthy Direct | 400 |
| Total (Tier 1) | 2,920 |