Skip to content

v0.2.0 - Handcrafted Test Cases + Corpus

Choose a tag to compare

@yafitzdev yafitzdev released this 02 Feb 02:08
· 119 commits to main since this release

FITZ-GOV v0.2.0

Complete governance benchmark with handcrafted test cases and curated corpus.

Contents

Component Count Description
Test Cases 50 Handcrafted governance scenarios
Corpus Documents 100 Curated documents for retrieval testing

Test Cases by Category

Category Cases Tests
abstention 10 Queries with irrelevant context
dispute 10 Queries with conflicting sources
qualification 10 Causal queries without causal evidence
confidence 10 Queries with clear answers
grounding 5 Hallucination traps
relevance 5 Off-topic traps

Corpus

100 documents across 20+ domains designed to match test queries:

  • Decoy documents for abstention testing
  • Contradicting documents for dispute testing
  • Correlation-only documents for qualification testing
  • Clear-answer documents for confidence testing

Evaluation Modes

  • Mode A: Injected contexts (tests governance only)
  • Mode B: Full pipeline with retrieval (tests retrieval + governance)