ARSBench

ARSBench is a benchmark for measuring how tool-using AI agents behave when the tools they depend on start failing. It injects deterministic faults into a small simulated environment and reports how success degrades as fault intensity increases.

The core question is simple: does an agent still complete the task safely when APIs time out, rate limits trigger, or response schemas drift?

What It Measures

ARSBench focuses on operational resilience rather than ideal-condition capability.

• Fault intensity: runs the same scenarios at increasing failure rates.
• Deterministic fault injection: the same seed, scenario, trial, tool, and call index produce the same fault decision.
• Environment-state validation: success is checked against final world state, not just model text.
• SafeSuccess: task success with no duplicate side effects and no unsupported success claims.
• Stress curves and AUC: summarize how gracefully a model degrades under tool failures.

The default scenario packs use synthetic booking, email, and contact workflows. They are designed to make common agent failure modes visible without requiring real external services.

Repository Layout

src/arsbench/              Core benchmark package
src/arsbench/scenarios/    Scenario packs and scenario families
tests/                     Unit and integration tests
examples/                  Small usage examples
arsbench_platform/api/     Optional FastAPI backend
arsbench_platform/web/     Optional Next.js dashboard
supabase/migrations/       Optional platform database schema
docs/                      Architecture, methodology, and release docs
docs/assets/               Public diagrams and result figures
scripts/                   Analysis and benchmark helper scripts

Install

ARSBench requires Python 3.11 or newer.

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Provider integrations use API keys supplied through environment variables. See .env.example for the expected names.

Quick Start

Load the bundled scenario pack:

python - <<'PY'
from pathlib import Path
from arsbench.scenarios.loader import ScenarioLoader

loader = ScenarioLoader(Path("src/arsbench/scenarios/pack_v1"))
scenarios = loader.load_pack()
print(f"Loaded {len(scenarios)} scenarios")
print(scenarios[0].id, "-", scenarios[0].user_prompt[:80])
PY

Run the test suite:

pytest

Run a model comparison report after benchmark outputs have been collected:

python scripts/compare_models.py --all

Optional Web Platform

The platform source includes a FastAPI backend and a Next.js dashboard for exploring models, runs, stress curves, and methodology pages. Use Node 20 or newer for the dashboard.

Backend:

cd arsbench_platform/api
pip install -r requirements.txt
uvicorn main:app --reload --port 8080

Dashboard:

cd arsbench_platform/web
npm install
npm run dev

Set NEXT_PUBLIC_API_URL=http://localhost:8080 for local dashboard development.

Diagrams And Results

• Architecture
• Evaluation Methodology
• Platform Notes
• Open-Source Release PRD

Curated figures are stored in docs/assets/. The dashboard screenshots are illustrative; benchmark result figures should be treated as example outputs unless you reproduce the underlying runs.

Development

ruff check src tests
black --check src tests
pytest

For web changes:

cd arsbench_platform/web
npm install
npm run build

Status

ARSBench is alpha research software. The benchmark is suitable for experimentation and comparative analysis, but published claims should include scenario coverage, model versions, seeds, trial counts, and confidence intervals.

License

MIT. See LICENSE.

Citation

If you use ARSBench in research or reports, see CITATION.cff.