🎯 AgentHarness

Advanced Evaluation Framework for AI Agents, Prompts, and Structured Outputs

Overview • Key Features • Architecture • Quick Start

---

📖 Overview

AgentHarness is a robust, research-grounded evaluation framework designed to bring rigorous reliability testing to LLM-based agents. We built this framework to move beyond basic correctness testing and introduce true statistical guarantees for AI agent behaviors in complex, multi-step environments.

Our platform supports full deterministic trajectory evaluations, tool-channel failure injections, statistical confidence intervals (Wilson/Bootstrap), and reproducible local SQLite caching out-of-the-box.

---

✨ Key Features

Our recent major enhancements (G1-G5) have transformed the system into a state-of-the-art evaluation benchmark tool:

• 🛡️ Versioned Agent Task Suite & Mocks: A deterministic registry of agent tasks (TaskRegistry) and a MockToolEnvironment that allows agent evaluation loops to run locally, intercepting tool calls for fast, reproducible testing.
• 📊 Statistical Aggregation Layer: In addition to simple means, the system calculates standard deviation, 95% Wilson Score Confidence Intervals (for pass rates), and Bootstrap CIs (for continuous metrics), proving reliability with mathematical rigor.
• ⏳ RPM-Aware Request Scheduler & Cache: Built-in BudgetScheduler respects LLM API RPM (Requests Per Minute) limits to prevent rate-limiting during massive evaluation batches. A persistent SQLite response cache ensures identical LLM prompts yield zero-latency cached responses, saving time and cost.
• 💾 SQLite Experiment Store & Reproducibility: Tracks every experiment run, seed, and content hash in a persistent .harness_experiments.db, allowing perfect deterministic replay and auditing of agent trajectories.
• 💥 Tool-Channel Failure-Injection: The ToolFailureInjector engine lets you inject timeouts, malformed JSON, and partial successes into the agent's environment to test recovery and robustness dynamically.

---

🏗️ System Architecture & Workflows

AgentHarness separates the concerns of execution, evaluation, and reporting into a highly modular pipeline.

flowchart TD
    %% Define Styles
    classDef layer fill:#1e1e1e,stroke:#74b9ff,stroke-width:2px,color:#dfe6e9,rx:10px,ry:10px;
    classDef db fill:#1e1e1e,stroke:#55efc4,stroke-width:2px,color:#dfe6e9,rx:10px,ry:10px;
    classDef logic fill:#1e1e1e,stroke:#a29bfe,stroke-width:2px,color:#dfe6e9,rx:10px,ry:10px;
    classDef fault fill:#1e1e1e,stroke:#ff7675,stroke-width:2px,color:#dfe6e9,rx:10px,ry:10px,stroke-dasharray: 5 5;

    subgraph Config [Test Configuration]
        A["TaskRegistry"]:::logic
        B["EvalConfig & Seed"]:::logic
    end

    subgraph Exec [1. Execution Layer]
        direction LR
        C["AgentTarget"]:::layer
        D["MockToolEnvironment"]:::layer
        E(("ToolFailureInjector")):::fault
        C <-->|"Execute Tools"| D
        E -.->|"Inject Faults<br/>(Timeouts, Bad Data)"| D
    end

    subgraph Eval [2. Evaluation Engine]
        direction LR
        F["Deterministic Metrics<br/>(ToolCorrectness)"]:::logic
        G["LLM-Judged Metrics<br/>(Faithfulness)"]:::logic
    end

    subgraph Agg [3. Statistical Aggregation]
        H["ScoreSummary"]:::layer
        I["Wilson CI & Bootstrap"]:::layer
        H --> I
    end

    subgraph Store [4. Storage & Reproducibility]
        J[("SQLiteExperimentStore")]:::db
    end

    %% Flows
    A --> Exec
    B --> Exec
    Exec -->|"Produces Trajectory"| Eval
    Eval -->|"Raw Scores"| Agg
    Agg -->|"Aggregated Results"| Store
    Exec -.->|"Saves Replay Data"| Store

Loading

1. Execution Layer (AgentTarget)

The execution layer drives the agent. Instead of running against live APIs (which are flaky and non-deterministic), agents run inside our MockToolEnvironment. The AgentTarget iteratively collects agent actions, executes them against mocked tools, and manages the trajectory state. Faults (such as timeouts or partial results) can be deterministically injected here via the ToolFailureInjector.

2. Evaluation Engine

Once a trajectory is generated, the Evaluation Engine applies Metrics. These metrics can be deterministic (e.g., ToolCorrectnessMetric, ToolArgumentMatchMetric) or LLM-judged (e.g., Faithfulness, AnswerRelevancy).

3. Statistical Aggregation (ScoreSummary)

Raw metric outputs are aggregated into the ScoreSummary. Instead of just providing a raw average, the framework calculates standard deviations and 95% Wilson Confidence Intervals. This gives engineers real statistical guarantees about agent reliability at scale.

4. Storage & Reproducibility (SQLiteExperimentStore)

Everything is saved locally to an SQLite database. By persisting the EvalConfig (with its seed and content hash), you can easily query past experiments, replay trajectories, and track benchmark regressions over time.

---

🚀 Quick Start

Installation

# Clone the repository
git clone https://github.com/Akgithub2028/Agent-Harness.git
cd Agent-Harness

# Create a virtual environment and install the framework
python3 -m venv venv
source venv/bin/activate
pip install -e .

Running a Benchmark

AgentHarness makes it easy to run automated benchmarks to test your agents against injected failures.

import asyncio
from agentharness.core.golden import Golden
from agentharness.core.runner import evaluate_dataset
from agentharness.metrics import ToolCorrectnessMetric, ToolArgumentMatchMetric

# Define your Goldens (Ground Truth)
goldens = [
    Golden(
        input="What is the weather in Paris?", 
        expected="Sunny, 18C", 
        expected_tools=["get_weather"], 
        expected_tool_calls=[{"name": "get_weather", "args": {"city": "Paris"}}]
    )
    for _ in range(10)
]

# Run the dataset evaluation against your Mock Agent
metrics = [ToolCorrectnessMetric(mode="exact"), ToolArgumentMatchMetric(arg_match="exact")]
results = asyncio.run(evaluate_dataset(goldens, your_mock_agent, metrics=metrics))

---

📈 Example Benchmark Output

AgentHarness outputs rigorous metrics summarizing performance:

=========================================
📊 AGENT EVAL BENCH: REAL METRIC OUTPUT 📊
=========================================
Sample Size (N)      : 10
Success Rate (P)     : 80.0% (8.0/10 successes)
95% Wilson CI        : [49.0%, 94.3%]
Outcome Variance     : 0.1600
Outcome Std Dev      : 0.4000
Trajectory Consistency: 70.0%
Avg Tool Selection   : 0.80
Avg Argument Match   : 0.80

These results highlight the viability of the statistical aggregation layer and the deterministic trajectory evaluations inside the framework.

---

📚 References & Enhancements

AgentHarness builds upon and enhances foundational concepts from leading academic and industry efforts in AI reliability:

• Towards a Science of AI Agent Reliability (Princeton, 2026): Defines 12 core reliability metrics. Our Enhancement: We added the G2 Statistical Aggregation Layer (Wilson/Bootstrap CIs) and G5 Tool-Channel Failure-Injection to measure these metrics empirically and deterministically.
• DeepEval & RAGAS: Popular frameworks for LLM and RAG evaluation. Our Enhancement: We extended their metric evaluation patterns with deterministic local task execution (G1: Agent Mocks) and state-of-the-art RPM scheduling (G3: BudgetScheduler) to prevent rate limits during massive dataset evaluations.
• promptfoo: A CLI-first eval tool that validated CI/CD-native approaches. Our Enhancement: We introduced G4: SQLite Experiment Store to provide robust persistence and perfect reproducibility across parallelized CI/CD test runs.

---

📄 License

This project is licensed under the Apache License, Version 2.0 - see the LICENSE file for details.