Repository navigation
v5.0.0: Grounding/relevance become cross-cutting quality checks
Highlights
Grounding & Relevance Are Now Cross-Cutting Quality Checks - Eliminated grounding and relevance as standalone categories. They are now quality dimensions applied to every trustworthy case (hedged and direct). Each trustworthy case now produces three scores: governance mode accuracy, grounding (did the response avoid hallucination?), and relevance (did the response address the question?). The benchmark drops from 6 categories to 4, with no data loss -- all 2,980 cases preserved.
Breaking Changes
- Removed
FitzGovCategory.GROUNDINGandFitzGovCategory.RELEVANCEenum values - 4 categories only: abstention, dispute, trustworthy_hedged, trustworthy_direct
grounding.jsonandrelevance.jsondata files no longer exist (merged intotrustworthy_hedged.json)FitzGovCaseResulthas new fields:mode_correct,grounding_passed,relevance_passed,grounding_failure,relevance_failureFitzGovCategoryResulthas new fields:grounding_accuracy,relevance_accuracy- Score comparisons between v4.1 and v5.0 are NOT directly comparable due to structural changes
Data Migration
- 676 grounding/relevance cases converted to trustworthy_hedged with prefixed subcategories (
grounding_*,relevance_*) - 884 existing trustworthy cases enriched with
forbidden_claimsandrequired_elementsannotations - All trustworthy cases (1,560 tier1 + 36 tier0) now have both quality annotation fields
- Trustworthy Hedged tier1: 484 → 1,160 cases (absorbed 336 grounding + 340 relevance)
- Trustworthy Direct: unchanged at 400 tier1 / 10 tier0
- All existing case IDs preserved
Evaluation Changes
- Unified evaluation flow: All categories use governance mode classification. Trustworthy categories additionally run grounding and relevance quality checks when mode is correct.
- Quality checks are conditional: If the system picks the wrong governance mode, quality checks are skipped
- 3-dimensional scoring for trustworthy categories
- No LLM-as-judge: Quality checks use regex-only validation (optional LLM validation still supported)
- All 4 categories participate in the confusion matrix
Distribution (Tier 1)
| Category | Cases | Medium | Hard | Med % |
|---|---|---|---|---|
| Trustworthy Hedged | 1,160 | 435 | 725 | 38% |
| Abstention | 685 | 255 | 430 | 37% |
| Dispute | 675 | 261 | 414 | 39% |
| Trustworthy Direct | 400 | 145 | 255 | 36% |
Migration Notes
pip install fitz-gov==5.0.0to upgrade- Remove any references to
FitzGovCategory.GROUNDINGorFitzGovCategory.RELEVANCE GOVERNANCE_MODE_CATEGORIESandANSWER_QUALITY_CATEGORIESconstants removed; useTRUSTWORTHY_CATEGORIESinstead- Case IDs unchanged -- grounding/relevance case IDs still work with
load_case_by_id() evaluate_case()now returns richer results with quality check fields
See CHANGELOG.md for full details including v4.1.0 changes.