Skip to content

v5.0.0: Grounding/relevance become cross-cutting quality checks

Choose a tag to compare

@yafitzdev yafitzdev released this 01 Mar 04:10
· 44 commits to main since this release

Highlights

Grounding & Relevance Are Now Cross-Cutting Quality Checks - Eliminated grounding and relevance as standalone categories. They are now quality dimensions applied to every trustworthy case (hedged and direct). Each trustworthy case now produces three scores: governance mode accuracy, grounding (did the response avoid hallucination?), and relevance (did the response address the question?). The benchmark drops from 6 categories to 4, with no data loss -- all 2,980 cases preserved.

Breaking Changes

  • Removed FitzGovCategory.GROUNDING and FitzGovCategory.RELEVANCE enum values
  • 4 categories only: abstention, dispute, trustworthy_hedged, trustworthy_direct
  • grounding.json and relevance.json data files no longer exist (merged into trustworthy_hedged.json)
  • FitzGovCaseResult has new fields: mode_correct, grounding_passed, relevance_passed, grounding_failure, relevance_failure
  • FitzGovCategoryResult has new fields: grounding_accuracy, relevance_accuracy
  • Score comparisons between v4.1 and v5.0 are NOT directly comparable due to structural changes

Data Migration

  • 676 grounding/relevance cases converted to trustworthy_hedged with prefixed subcategories (grounding_*, relevance_*)
  • 884 existing trustworthy cases enriched with forbidden_claims and required_elements annotations
  • All trustworthy cases (1,560 tier1 + 36 tier0) now have both quality annotation fields
  • Trustworthy Hedged tier1: 484 → 1,160 cases (absorbed 336 grounding + 340 relevance)
  • Trustworthy Direct: unchanged at 400 tier1 / 10 tier0
  • All existing case IDs preserved

Evaluation Changes

  • Unified evaluation flow: All categories use governance mode classification. Trustworthy categories additionally run grounding and relevance quality checks when mode is correct.
  • Quality checks are conditional: If the system picks the wrong governance mode, quality checks are skipped
  • 3-dimensional scoring for trustworthy categories
  • No LLM-as-judge: Quality checks use regex-only validation (optional LLM validation still supported)
  • All 4 categories participate in the confusion matrix

Distribution (Tier 1)

Category Cases Medium Hard Med %
Trustworthy Hedged 1,160 435 725 38%
Abstention 685 255 430 37%
Dispute 675 261 414 39%
Trustworthy Direct 400 145 255 36%

Migration Notes

  • pip install fitz-gov==5.0.0 to upgrade
  • Remove any references to FitzGovCategory.GROUNDING or FitzGovCategory.RELEVANCE
  • GOVERNANCE_MODE_CATEGORIES and ANSWER_QUALITY_CATEGORIES constants removed; use TRUSTWORTHY_CATEGORIES instead
  • Case IDs unchanged -- grounding/relevance case IDs still work with load_case_by_id()
  • evaluate_case() now returns richer results with quality check fields

See CHANGELOG.md for full details including v4.1.0 changes.