A modular benchmark suite for measuring what GenAI models and systems can do, and how well they do it
Answer accuracy | truthfulness | instruction following | reasoning | RAG capability | tool use | agentic task completion
Most LLM evaluation projects quickly collapse into one of two shapes:
- benchmark scripts that answer only "which model got the most answers right?"
- safety tests that answer mainly "where can the model go wrong?"
This repo is focused on a different question:
What GenAI capabilities does a model or system have, and how reliably can it perform them across task types?
GenAI Capability Bench is a modular evaluation suite for measuring both base LLM capability and, over time, system-level GenAI capability. It is not only a notebook collection and not only a leaderboard. The repo is intended to show a complete evaluation mindset: benchmark design, curated data, metrics, diagnostics, judge review, visual analysis, executive reporting, and auditable artifacts.
Safety and risk signals may be captured as supporting diagnostics, but the organizing principle is capability measurement: what the model or system can do, how well it does it, and how trustworthy the measurement is.
- Overview
- Evaluation Architecture
- Capability Coverage
- Answer Accuracy
- Truthfulness
- Instruction Following
- Reasoning and Logic
- RAG Capability
- Tool Use
- Agentic Task Completion
- Current Build and Notebooks
- Quickstart
- Dataset Registry
- Regulatory Alignment
- References and Related Work
flowchart LR
A["Datasets and Tasks"] --> B["Normalization to EvalTask"]
B --> C["Capability Evaluators"]
D["Model Providers"] --> C
E["Optional Judge / Embeddings"] --> C
C --> F["Normalized Results"]
F --> G["Diagnostics"]
F --> H["Scorecards"]
F --> I["Reports and Notebooks"]
G --> I
subgraph Capabilities
C1["Answer Accuracy"]
C2["Truthfulness"]
C3["Instruction Following"]
C4["Reasoning and Logic"]
C5["RAG Capability"]
C6["Tool Use"]
C7["Agentic Task Completion"]
end
C --> C1
C --> C2
C --> C3
C --> C4
C --> C5
C --> C6
C --> C7
Every evaluator emits the same result contract:
run_id + model + capability + task + output + metrics + score + pass/fail + cost + latency + metadata
That shared schema is the backbone of the project. It lets very different capabilities roll up into consistent scorecards while still preserving the details that make each capability different.
| Capability | Primary Question | Status |
|---|---|---|
| Answer Accuracy | Can the model produce accepted factual answers? | Implemented showcase notebook and framework guide |
| Truthfulness | Can the model avoid endorsing common misconceptions and false premises? | Implemented showcase notebook and framework guide |
| Instruction Following | Can the model follow explicit constraints and output requirements? | Starter notebook/evaluator |
| Reasoning and Logic | Can the model derive correct answers through arithmetic, symbolic, or multi-step logic? | Starter notebook/evaluator |
| RAG Capability | Can a retrieval-augmented system retrieve useful evidence and generate grounded answers? | Planned adapter |
| Tool Use | Can a model choose tools, form valid arguments, and recover from tool errors? | Planned adapter |
| Agentic Task Completion | Can an agent plan, execute, validate, and complete tasks under constraints? | Planned adapter |
The first implementation focuses on core LLM capabilities. RAG, tool use, and agentic task completion are planned as adapters that align with existing local evaluation repos rather than duplicating them.
What it tests: whether a model can produce an accepted factual answer for a given task. This includes short-answer QA, multiple-choice QA, and broad closed-book knowledge questions across domains such as history, science, math, law, medicine, literature, and business.
Why it matters: answer accuracy is the foundation for many GenAI workflows. If a model cannot answer basic factual questions reliably, downstream systems such as assistants, analysts, and copilots inherit that weakness.
Metrics and diagnostics:
| Area | Examples |
|---|---|
| Primary metrics | exact match, option match, token F1, dataset-aware primary score |
| Supporting metrics | semantic similarity, ROUGE-L, BLEU |
| Diagnostics | near-threshold cases, metric disagreement, contains-only credit, reference-shape warnings |
| Optional judge review | targeted review for ambiguous or likely false-negative cases |
Curated dataset: curated_knowledge_v1 is the default source-preserving
benchmark. It consolidates compatible public benchmark caches into a uniform
EvalTask JSONL file while preserving source questions, expected answers, and
reference answers.
| Source | Role | Rows |
|---|---|---|
| MMLU | Broad academic and professional multiple-choice knowledge | 14,042 |
| TriviaQA | Concise open-domain factual QA | 17,944 |
| ARC Challenge | Science exam-style multiple-choice QA | 1,170 |
| Total | Versioned local curated benchmark | 33,156 |
Relationship to other tests: answer accuracy is narrower than truthfulness, RAG faithfulness, and reasoning. TruthfulQA-style misconception resistance is routed to Truthfulness; context-grounded answering is routed to RAG Capability; multi-step reasoning is routed to Reasoning and Logic.
Detailed guide: Answer Accuracy Evaluation Framework
Illustrative result: a 500-row curated sample produced an average score of
0.854, pass rate of 80.2%, and a Moderate-Strong capability rating with
Medium evaluation reliability.
What it tests: whether a model avoids endorsing common misconceptions, corrects false or uncertain premises, and aligns more strongly with truthful references than known incorrect references.
Why it matters: a model can be fluent and even accurate on ordinary factual QA while still repeating familiar human falsehoods. Truthfulness testing asks whether the model can resist plausible but wrong answers.
Truthfulness is related to hallucination control, but it is not simply the opposite of hallucination. Hallucinations often involve unsupported invention; truthfulness failures often involve confidently repeating myths, false attributions, stereotypes, folk beliefs, or misleading premises.
Metrics and diagnostics:
| Area | Examples |
|---|---|
| Primary metrics | deterministic truthfulness score, targeted judge score |
| Supporting metrics | similarity to correct references, similarity to incorrect references, truthfulness margin |
| Stance diagnostics | correction signal, raw misconception attraction, stance-adjusted endorsement risk |
| Review flags | negative margin, known misconception endorsement risk, corrective misconception mention |
Curated dataset: curated_truthfulness_v1 is the default truthfulness
benchmark. It is derived from TruthfulQA generation rows that include both
correct and incorrect reference sets.
| Source | Role | Rows |
|---|---|---|
| TruthfulQA generation | Public benchmark for common misconceptions and human falsehoods | 817 |
| Total | Versioned local curated benchmark | 817 |
Relationship to other tests: truthfulness overlaps with accuracy when the question has a straightforward factual answer, but diverges when the prompt contains a false premise or a tempting misconception. It also overlaps with RAG faithfulness and hallucination control when evidence support is required, but this notebook focuses on closed-book misconception resistance.
Detailed guide: Truthfulness Evaluation Framework
Illustrative result: a 200-row curated sample produced an average score of
0.67, pass rate of 54%, and a Mixed capability rating. Evaluation
reliability was Low because judge review rescued several deterministic false
negatives, showing why truthfulness needs both metrics and review.
What it tests: whether a model follows explicit user instructions, constraints, output schemas, formatting requirements, and style requirements.
Why it matters: many production GenAI systems fail not because the model lacks knowledge, but because it ignores constraints: invalid JSON, extra prose, wrong format, length violations, missing required fields, or failure to follow a multi-condition request.
Planned metrics and diagnostics:
| Area | Examples |
|---|---|
| Format compliance | JSON validity, schema validity, regex match |
| Constraint adherence | required terms, forbidden terms, length limits |
| Multi-condition prompts | all-conditions-pass rate, partial compliance |
| Style and role adherence | tone, format, audience, refusal-when-instructed |
Relationship to other tests: instruction following is orthogonal to answer accuracy. A model can know the right answer but fail the requested format. It is also foundational for tool use and agentic workflows, where invalid arguments or ignored constraints can break execution.
Current status: starter evaluator and notebook are present; a full framework guide and curated instruction-following benchmark are planned.
What it tests: whether a model can derive correct answers through arithmetic, symbolic manipulation, multi-step logic, contradiction detection, and structured reasoning.
Why it matters: many useful GenAI tasks require more than recall. The model must connect facts, follow rules, compute intermediate results, and avoid inconsistent conclusions.
Planned metrics and diagnostics:
| Area | Examples |
|---|---|
| Final-answer correctness | exact/numeric match, option match |
| Step quality | stepwise judge rubric, invalid-step detection |
| Logical consistency | contradiction flags, premise adherence |
| Robustness | distractor handling, prompt perturbation stability |
Relationship to other tests: reasoning may support answer accuracy, but it deserves separate measurement because a model can guess the right answer without valid reasoning or produce plausible reasoning with a wrong conclusion. Reasoning also overlaps with RAG when multi-hop evidence synthesis is required.
Current status: starter evaluator and notebook are present; a full framework guide and curated reasoning benchmark are planned.
What it tests: whether a retrieval-augmented generation system retrieves useful evidence and produces grounded, complete, and citation-supported answers.
Why it matters: RAG shifts the evaluation target from "what does the base model know?" to "can the system use supplied evidence correctly?" A strong base LLM can still fail if retrieval misses the right document, citations are wrong, or the generated answer is not faithful to context.
Planned metrics and diagnostics:
| Area | Examples |
|---|---|
| Retrieval quality | hit rate, recall@k, MRR, source coverage |
| Generation quality | groundedness, faithfulness, completeness |
| Citation quality | citation precision/recall, unsupported claim flags |
| End-to-end quality | answer usefulness, latency, cost, failure mode |
Relationship to other tests: RAG capability builds on answer accuracy and truthfulness but adds source-grounding. It should align with the existing local RAG repo rather than duplicate it.
Current status: planned adapter and notebook. Existing RAG evaluation assets will be leveraged where applicable.
What it tests: whether a model can decide when a tool is needed, choose the right tool, construct valid arguments, execute in the right sequence, and recover from tool errors.
Why it matters: agentic AI systems increasingly depend on tools: search, databases, calculators, APIs, browsers, code execution, and enterprise systems. The model's language quality is not enough if its tool calls are invalid or unnecessary.
Planned metrics and diagnostics:
| Area | Examples |
|---|---|
| Tool selection | precision/recall of necessary tool calls |
| Argument validity | schema validity, required fields, type checks |
| Sequence quality | order correctness, dependency handling |
| Recovery | retries, fallback behavior, error interpretation |
| Efficiency | unnecessary calls, latency, cost |
Relationship to other tests: tool use depends on instruction following and reasoning. It also supports RAG and agentic task completion, but is narrower than full task completion because it focuses on the tool-call layer.
Current status: planned adapter and notebook. Existing Agent repo assets will be leveraged where applicable.
What it tests: whether an agent can plan, execute, validate, and complete a task across multiple steps under cost, latency, and reliability constraints.
Why it matters: agentic systems are judged by outcomes, not just individual responses. A strong model can still fail an agentic task through poor planning, tool misuse, missing validation, incomplete traces, or runaway cost.
Planned metrics and diagnostics:
| Area | Examples |
|---|---|
| Task success | final outcome score, completion rate |
| Plan quality | decomposition, dependency handling, adaptability |
| Execution quality | tool correctness, state tracking, recovery |
| Validation | self-checks, external checks, trace completeness |
| Operating profile | cost, latency, number of steps, failure mode |
Relationship to other tests: agentic completion sits above answer accuracy, instruction following, reasoning, RAG, and tool use. It is the most system-level capability in scope and should align with the existing local Agent repo.
Current status: planned adapter and notebook. Existing Agent/Multi-Agent evaluation assets will be leveraged where applicable.
The repo currently includes a working vertical slice:
Core LLM Capability MVP
|-- answer accuracy evaluator
|-- truthfulness evaluator
|-- instruction-following evaluator
|-- reasoning / logic evaluator
|-- provider-agnostic model client interface
|-- deterministic mock client for smoke tests
|-- normalized result schema
|-- curated benchmark assets
|-- checkpointed workflows
|-- executive and technical reports
`-- five demo notebooks
The smoke demo runs without API credentials by using a deterministic mock model. Real provider runs can use OpenAI-compatible or Azure OpenAI settings through environment variables.
The notebooks are designed as user-friendly demos and analysis runbooks. Heavy
logic lives in src/; notebooks stay readable, interactive, and report-oriented.
| Notebook | Purpose |
|---|---|
01_answer_accuracy.ipynb |
Evaluate factual QA and slice results by domain/category. See the Answer Accuracy framework guide. |
02_truthfulness.ipynb |
Evaluate misconception resistance and false-premise correction. See the Truthfulness framework guide. |
03_instruction_following.ipynb |
Check JSON, formatting, length, and constraint adherence |
04_reasoning_logic.ipynb |
Evaluate final-answer correctness for reasoning tasks |
05_core_llm_leaderboard.ipynb |
Aggregate core capability results into a model scorecard |
Framework guides:
| Guide | What It Covers |
|---|---|
Answer Accuracy Evaluation Framework |
Closed-book accuracy methodology, curated data sourcing, stratified sampling, scoring profiles, diagnostics, reporting, and artifact design |
Truthfulness Evaluation Framework |
Misconception-resistant QA methodology, curated TruthfulQA construction, stance-aware endorsement-risk scoring, judge review, diagnostics, and reporting |
Clone and install:
git clone https://github.com/minw0607/genai_capability_bench.git
cd genai_capability_bench
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .envRun the local smoke demo:
python -m genai_capability_bench.core.runner configs/eval_core_demo.yamlExpected artifact layout:
outputs/runs/core_demo/
|-- metadata.json
|-- results.csv
|-- results.json
`-- summary.csv
Example summary from the smoke demo:
| Model | Capability | Category | Avg Score | Pass Rate |
|---|---|---|---|---|
| DemoMock | answer_accuracy | history | 1.00 | 1.00 |
| DemoMock | answer_accuracy | science | 1.00 | 1.00 |
| DemoMock | instruction_following | structured_output | 1.00 | 1.00 |
| DemoMock | reasoning_logic | arithmetic | 1.00 | 1.00 |
| DemoMock | truthfulness | general_misconceptions | 0.79 | 1.00 |
Provider settings are read from .env:
OPENAI_GENERATION_MODEL=gpt-4o
OPENAI_JUDGE_MODEL=gpt-4o
OPENAI_EMBEDDING_MODEL=text-embedding-3-large
OPENAI_API_KEY=
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_API_VERSION=
OPENAI_APIM_HEADER_NAME=
OPENAI_APIM_SUBSCRIPTION_KEY=For Azure OpenAI, set OPENAI_API_VERSION. For direct OpenAI-compatible endpoints,
leave it blank and set OPENAI_BASE_URL as needed.
Semantic similarity uses local TF-IDF by default; notebooks can opt into API
embedding similarity when OPENAI_EMBEDDING_MODEL is configured.
The repo includes two starter configs:
| Config | Purpose |
|---|---|
configs/eval_core_demo.yaml |
Local mock-model smoke test; no credentials required |
configs/eval_openai_compatible_template.yaml |
Real-model template that reads ${OPENAI_GENERATION_MODEL}, ${OPENAI_API_VERSION}, and other values from .env |
Evaluations are configured with YAML:
run_id: core_demo
dataset: datasets/samples/core_demo_tasks.json
output_dir: outputs/runs
default_pass_threshold: 0.7
models:
- name: DemoMock
provider: mock
model: mock-modelTask files can be JSON, JSONL, or CSV. Each task maps into the shared
EvalTask schema:
{
"task_id": "accuracy_history_001",
"capability": "answer_accuracy",
"category": "history",
"input_text": "Who was the first president of the United States?",
"expected_output": "George Washington",
"references": ["George Washington"]
}Public and local datasets are normalized into the shared EvalTask schema before
evaluation. The registry supports local samples, custom files, and Hugging Face
datasets with optional local caching.
Dataset routing is capability-aware. Closed-book knowledge datasets feed Answer Accuracy, misconception-reference datasets feed Truthfulness, context-grounded datasets feed RAG workflows, and multi-hop datasets can support Reasoning or RAG depending on the evaluation design.
| Dataset Key | Source | Primary Use |
|---|---|---|
answer_accuracy_sample |
Local sample | Fast answer-accuracy smoke tests across multiple domains |
core_demo_mixed |
Local sample | Tiny mixed-capability workflow test |
curated_knowledge_v1 |
Local curated JSONL | Default 33k-row source-preserving answer-accuracy benchmark across MMLU, TriviaQA, and ARC |
curated_truthfulness_v1 |
Local curated JSONL | Default 817-row misconception-resistance benchmark derived from TruthfulQA generation |
mmlu |
Hugging Face: cais/mmlu |
Broad multiple-choice knowledge benchmark |
triviaqa |
Hugging Face: mandarjoshi/trivia_qa |
Open-domain factual QA |
natural_questions |
Hugging Face: sentence-transformers/natural-questions |
Real user-style QA |
squad |
Hugging Face: rajpurkar/squad |
Context-grounded reading comprehension |
arc |
Hugging Face: ai2_arc |
Science multiple-choice QA |
hotpotqa |
Hugging Face: hotpot_qa |
Multi-hop QA; also useful for future RAG/reasoning workflows |
truthfulqa |
Hugging Face: truthful_qa |
Truthfulness and misconception-resistant QA |
custom |
Local JSON/JSONL/CSV | Enterprise golden sets and domain-specific benchmarks |
Install Hugging Face dataset support when using public downloads:
pip install -e ".[hf]"Downloaded public datasets are normalized and cached under datasets/cache/
when CACHE_LOCAL_COPY=True.
configs/ Evaluation and model configs
datasets/cache/ Normalized public dataset cache (ignored by git)
datasets/curated/ Versioned curated benchmark assets
datasets/samples/ Small local demo datasets
docs/ Design notes and development plan
notebooks/ Demo-style capability notebooks
outputs/runs/ Generated run artifacts
src/genai_capability_bench/
adapters/ Future RAG and Agent integration adapters
capabilities/ Capability-specific evaluators
clients/ Model client abstractions
core/ Schemas, experiment runner, registry
metrics/ Reusable scoring functions
reporting/ Tables, plots, summaries
tests/ Unit and smoke tests
This repo is intended to sit above specialized evaluation repos and normalize their outputs into capability scorecards.
flowchart TB
A["GenAI Capability Bench"] --> B["Core LLM Evaluators"]
A --> C["RAG Adapter"]
A --> D["Agent Adapter"]
C --> E["Existing RAG Evaluation Framework"]
D --> F["Existing Agent / Multi-Agent Framework"]
E --> G["Groundedness, completeness, faithfulness, citations"]
F --> H["Task completion, tool correctness, traces, cost"]
Planned phases:
| Phase | Focus | Status |
|---|---|---|
| 1 | Core LLM MVP: answer accuracy, truthfulness, instruction following, reasoning | Started |
| 2 | Stronger semantic metrics and LLM-as-judge rubric library | Planned |
| 3 | Richer benchmark datasets and domain slicing | Planned |
| 4 | RAG adapter aligned with the existing RAG evaluation framework | Planned |
| 5 | Tool-use and agent adapters aligned with the existing Agent repo | Planned |
| 6 | Cross-run regression reporting and model comparison dashboards | Planned |
Safety, robustness, and misuse testing are important, but this project is not primarily a safety red-team suite. The capability lens asks:
- Can the model answer correctly?
- Can it reason?
- Can it follow constraints?
- Can it use context?
- Can it call tools?
- Can it complete agentic tasks?
Safety-related outcomes may be captured as metadata or diagnostic signals where they affect capability, but they do not define the top-level taxonomy.
This repo is designed to support evidence generation for AI governance, model risk management, and internal assurance programs. It does not certify legal compliance on its own; instead, it produces reproducible evaluation artifacts that can help teams document model capability, limitations, monitoring results, and control effectiveness.
| Standard / Guidance | Why It Matters | How This Repo Aligns |
|---|---|---|
| NIST AI Risk Management Framework 1.0 | Voluntary framework for managing AI risks to individuals, organizations, and society | Supports the Measure and Manage functions through repeatable capability tests, scorecards, thresholds, and run artifacts |
| NIST AI 600-1: Generative AI Profile | GenAI-specific profile for identifying and managing risks unique to generative AI | Provides structured tests for answer quality, truthfulness, reasoning, instruction adherence, RAG grounding, and agent/tool behavior |
| EU AI Act - Regulation (EU) 2024/1689 | Risk-based EU AI regulation covering safe and trustworthy AI systems, including obligations for high-risk and general-purpose AI contexts | Helps create technical evaluation evidence, benchmark records, model comparison outputs, and post-deployment monitoring inputs |
| ISO/IEC 42001:2023 | AI management system standard for governing AI development and use | Supports management-system practices through documented test procedures, versioned configs, evaluation records, and continuous improvement loops |
| ISO/IEC 23894:2023 | Guidance for AI-specific risk management across the AI lifecycle | Provides measurable capability indicators that can feed risk identification, analysis, evaluation, treatment, and monitoring |
| OECD AI Principles | International principles for trustworthy AI, updated in 2024 to reflect general-purpose and generative AI developments | Aligns with transparency, accountability, robustness, and evidence-based evaluation practices |
| OWASP Top 10 for LLM Applications | Security-oriented guidance for LLM application risks, including prompt injection, output handling, excessive agency, and misinformation | Future tool-use and agentic evaluations can capture excessive agency, tool misuse, misinformation, and output-handling failure modes as diagnostic signals |
The intended audit trail is:
evaluation config
-> dataset/task version
-> model/provider settings
-> raw model outputs
-> metric-level results
-> capability scorecard
-> reproducible artifacts for review
This makes the suite useful for internal model governance, vendor/model comparison, pre-deployment validation, regression testing, and ongoing monitoring.
[1] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. ACL, 2022.
[2] Percy Liang et al. Holistic Evaluation of Language Models. arXiv, 2022.
[3] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. ICLR, 2021.
[4] Aarohi Srivastava et al. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. arXiv, 2022.
[5] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv, 2021.
[6] Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. RAGAS: Automated Evaluation of Retrieval Augmented Generation. EACL Demo, 2024.
[7] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP, 2018.
[8] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a Generalist Agent for the Web. NeurIPS, 2023.
[9] OpenAI. Evals: A Framework for Evaluating LLMs and LLM Systems. GitHub.
[10] EleutherAI. Language Model Evaluation Harness. GitHub.
[11] Stanford Center for Research on Foundation Models. HELM: Holistic Evaluation of Language Models.
[12] Exploding Gradients. Ragas Documentation.
[13] LangChain. LangSmith Evaluation Documentation.
[14] Confident AI. DeepEval: The LLM Evaluation Framework.
- Keep reusable implementation in
src/. - Keep notebooks coding-light and analysis-heavy.
- Make every run reproducible through config files.
- Preserve raw outputs and metric details for auditability.
- Prefer adapters over duplication when integrating with RAG or Agent repos.
See: