Skip to content

FITZ-GOV v0.1.0

Choose a tag to compare

@yafitzdev yafitzdev released this 01 Feb 23:24
· 122 commits to main since this release

FITZ-GOV Benchmark v0.1.0

First release of the FITZ-GOV governance benchmark.

Contents

  • 66,416 corpus documents from BEIR (scifact, nfcorpus, fiqa)
  • 175 test cases across 6 governance categories:
    Category Cases Tests
    abstention 28 Should refuse to answer
    dispute 30 Should flag conflicts
    qualification 28 Should hedge uncertain claims
    confidence 30 Should answer confidently
    grounding 29 No hallucination
    relevance 30 Stays on topic

Usage with fitz-ai

from fitz_ai.evaluation.benchmarks import FitzGovBenchmark

benchmark = FitzGovBenchmark()  # Auto-downloads this release
results = benchmark.evaluate(engine)

Manual Download

Download fitz_gov_data.zip and extract to ~/.fitz/fitz_gov_data/