Project brief

Summary

agent-bench-stability: a procedurally generated, programmatically verified agentic task environment plus a perturbation study that quantifies, with confidence intervals, how stable LLM-agent benchmark scores and model rankings are under changes that should not matter (prompt paraphrase, tool renaming, distractor tools, instance resampling, decoding temperature, task order). Built for a single engineer-researcher (Brian) as a public research artifact; the deliverables are a reusable environment/harness package and the result tables and figures for a write-up titled "How reliable is an agent benchmark?"

Problem

Agent benchmarks are used to pick models and to sell eval services, but a single score hides its own fragility: nobody publishing a leaderboard says how much the number moves when the prompt is reworded or a tool is renamed. Today, testing that requires bespoke scripts per model and per perturbation, with no guarantee that the agent scaffold is held constant or that reward checking is objective. Once this exists, one command materializes a pre-registered grid of (model x task family x perturbation) cells, runs each agent episode through an identical scaffold against seeded task instances with programmatic verifiers, and emits score distributions, rank-stability statistics, and a minimum-task- count power analysis that says how many tasks a benchmark needs before a claimed model difference is real.

Scope

• A task environment package with three procedurally generated, programmatically verified task families, each seeded and parameterized by difficulty:
  • tabular: transform a generated CSV to a generated spec (filter, join, aggregate); verifier compares output bytes to the known-correct result.
  • filesys: multi-step goals in a simulated in-memory file system with read, write, list, search tools; verifier checks final state against the generated goal condition.
  • query: natural-language questions against a generated SQLite database; verifier compares the agent's final answer to the computed ground truth.
• One fixed agent scaffold: a tool-calling loop with a step cap, identical system prompt template, and identical tool schemas for every model; model access through a single adapter supporting Anthropic and OpenAI-compatible chat/tool APIs, selected by config.
• Perturbation modules, each seedable and each provably ground-truth-preserving:
  • Paraphrase: 5 pre-written instruction paraphrase templates per task family.
  • Tool renaming: systematic renaming of tool names/argument names.
  • Distractors: injection of 2 or 5 well-described but irrelevant tools.
  • Instance resampling: same family/difficulty, new seeds.
  • Decoding: temperature 0.0 vs 0.7.
  • Ordering: task presentation order shuffled per seed (for context-carrying harness modes; independent-episode mode is the default and documents this axis as a no-op).
• A run ledger recording per-episode token counts and API cost, with a hard per-run cost cap that halts new episodes when reached.
• Analysis: per-cell pass rates with bootstrap CIs; per-axis score deltas; cross-model rank stability (Kendall's tau across perturbations); variance decomposition (model vs. perturbation vs. instance); a power analysis table reporting the minimum number of task instances needed to detect 3/5/10-point pass-rate differences at 95% confidence; all figures as PNG and SVG with underlying data as Parquet.
• HYPOTHESES.md: pre-registered directional hypotheses per axis, committed before any result exists.
• A --smoke mode that runs the entire pipeline offline against a built-in deterministic scripted agent (no network, no API keys), exercising every task family and every perturbation, finishing in under 10 minutes on CPU.

Non-goals

• No model training, fine-tuning, or RLHF; models are evaluated as-is via API.
• No GPU requirement anywhere.
• No human grading or subjective rubrics; if a task cannot be verified programmatically it is out of scope.
• No live-web, browsing, or code-execution-in-sandbox tasks.
• No public leaderboard site or hosted service; outputs are files.
• No safety red-teaming or jailbreak testing.
• No claims about which model is best; the artifact is about measurement reliability, and the write-up prose itself is out of scope for the pipeline.

Functional requirements

ID	Requirement	Priority
R-001	generate materializes seeded task instances for each family/difficulty per the manifest, each with its ground truth and verifier inputs stored alongside	Must
R-002	For a fixed (manifest, seed), generated instances, perturbed variants, and verifier verdicts are bit-reproducible on the same platform	Must
R-003	perturb derives perturbed variants of an instance set without mutating the source set, and records which axis/level produced each variant	Must
R-004	run executes agent episodes for a (model, cell) through the fixed scaffold, persisting every episode transcript (messages, tool calls, results) as JSONL	Must
R-005	score applies the family verifier to each episode and appends one row per episode to results/results.parquet; verifiers read transcripts and final states only, never the model name	Must
R-006	analyze produces pass-rate tables with bootstrap CIs, per-axis deltas, Kendall-tau rank stability, variance decomposition, the power-analysis table, and all figures from results.parquet alone	Must
R-007	Every stage is resumable: re-running skips episodes whose completed transcripts and scores exist, and finishes interrupted cells	Must
R-008	The full grid (models, families, difficulties, axes, levels, instances per cell, seeds) is defined in one versioned manifest (experiments.yaml); no cell parameters live in code	Must
R-009	A per-run cost cap from the manifest halts new episodes when projected spend reaches the cap, marks the run partial, and exits 0 with an explicit summary	Must
R-010	--smoke runs generate, perturb, run, score, and analyze end-to-end offline with the scripted agent in ≤10 min on CPU	Must
R-011	Every results row records git SHA, manifest hash, seed, model identifier, adapter version, token counts, and cost	Must
R-012	analyze emits a per-hypothesis verdict table keyed to HYPOTHESES.md IDs (supported / refuted / inconclusive with CI bounds)	Should
R-013	A transcript viewer subcommand renders one episode's transcript as readable markdown for spot audits	Should
R-014	The environment package is importable and documented for standalone use (tasks + verifiers without the study harness)	Could

User-visible behavior

ID	Trigger	Expected result	On failure
B-001	agent-bench generate --manifest experiments.yaml	Instance sets written under data/instances/, summary printed (family, difficulty, count, seed)	Non-zero exit naming the family and parameter that failed validation; no partial set marked complete
B-002	agent-bench run --model <id> --cell <id> with no API key configured for that model	Refuses before any episode; names the missing environment variable	Exit 2, nothing written
B-003	Interrupting run mid-cell, then re-running the same command	Completed episodes are skipped with one log line each; the interrupted episode is discarded and re-run	—
B-004	Cost cap reached mid-run (R-009)	Run stops starting new episodes, prints episodes completed vs. planned and spend vs. cap, exits 0; analyze later lists the cell as partial	—
B-005	An API call fails or times out during an episode	The episode retries per the manifest's retry policy; if exhausted, the episode is recorded as infra_error (excluded from pass rates, counted in the run summary), and the run continues	—
B-006	agent-bench analyze with fewer completed cells than the manifest defines	Produces outputs for completed cells, prints an explicit list of missing/partial cells, exits 0 with a warning	—
B-007	Running any subcommand with --smoke	Same code paths against the scripted agent, no network; prints SMOKE OK with per-stage timings	Non-zero exit naming the first failed stage

Domain rules and invariants

ID	Invariant	Consequence if violated
I-001	Perturbations never change a task's ground truth or its verifier verdict for a correct solution; each perturbation module ships a property test asserting this on generated instances	Score deltas measure broken tasks, not model sensitivity
I-002	Every model in a cell runs the byte-identical scaffold, prompts (modulo the perturbation under test), tool schemas, step cap, and retry policy	Cross-model comparisons invalid
I-003	Verifiers are pure functions of transcript and final state; nothing model-identifying reaches them	Scoring bias
I-004	HYPOTHESES.md is committed before the first analyze output exists and is never edited by the pipeline	Pre-registration claim is false
I-005	A results row is written only after its verifier verdict is final; no partial rows	Analysis computed from garbage
I-006	Instance sets are immutable after generate; perturb writes only new variant directories	Silent cross-contamination
I-007	infra_error episodes are never counted as failures in pass rates and never silently dropped from the run summary	Reliability conclusions confounded by infrastructure noise

Data and state

Authoritative state is the filesystem: data/instances/ (immutable generated sets + ground truth), data/variants/<cell-id>/ (perturbed sets), runs/<model>/<cell-id>/episodes/*.jsonl (transcripts), results/results.parquet (append-only scored rows), results/analysis/ (tables, figures). experiments.yaml and HYPOTHESES.md are versioned in git; data/, runs/, and results/ are gitignored except results/analysis/. Episode completion is a _COMPLETE marker written after the transcript is flushed; a transcript without the marker is treated as interrupted and re-run. API keys live only in environment variables and are never written to any artifact, transcript, or log. Everything on disk survives restart; nothing is held only in memory.

Interfaces

• CLI: single entry point agent-bench with subcommands generate, perturb, run, score, analyze, view (R-013), each accepting --manifest (default experiments.yaml), --cell/--model where applicable, --seed, and --smoke.
• experiments.yaml: declares models (adapter type, model id, env-var name for the key, pricing per token for the ledger), task families with difficulty parameters and instances-per-cell, perturbation axes and levels, seeds, step cap, retry policy, and the cost cap. Schema owned by this project, validated on load with actionable errors.
• Model adapter interface: complete(messages, tools, temperature) -> (message, tool_calls, usage); implementations for Anthropic and OpenAI-compatible APIs, plus the scripted smoke agent. New providers are added by implementing this one interface.
• results/results.parquet schema (one row per episode): episode_id, model, family, difficulty, axis, level, instance_seed, run_seed, passed, partial_score, steps_used, infra_error, tokens_in, tokens_out, cost_usd, git_sha, manifest_hash, adapter_version, wall_clock_s.
• External services: only the model APIs named in the manifest, only during run.

Constraints

• Python 3.11+; pydantic for manifest/schema validation, httpx for API adapters, pandas+pyarrow for results, matplotlib for figures, scipy for tau/bootstrap/power computations, sqlite3 from the standard library for the query family. No proprietary dependencies; no GPU.
• Full smoke run CPU-only and offline (macOS included). Real runs need only API keys and network to the configured endpoints.
• Default shipped grid must be executable for under $200 in API spend at the manifest's pricing entries, enforced by the cost cap.
• Disk budget ≤5 GB total.
• Repository is intended to be public: no credentials in history, no absolute local paths in committed files, license file included.

Failure behavior

Invalid manifest: fail at load with field path and expected type, before any work. Instance generation failure: fail that set, leave nothing marked complete. API failures: retry per policy, then infra_error per B-005; an entire cell whose episodes are all infra_error is flagged loudly in the run summary and by analyze. Interrupted runs: resume per B-003. Concurrent invocation on the same cell: lock file, second invocation refused with a clear message. Cost cap: per B-004, always a visible partial, never a silent stop. Out-of-disk: fail loudly with the path; the _COMPLETE protocol guarantees partials are re-run. A verifier exception (as opposed to a false verdict) is recorded as infra_error, never as a model failure.

Verification

• ruff check and ruff format --check clean; mypy clean on the package.
• pytest suite covering: manifest validation; determinism of generation and perturbation (same seed ⇒ identical bytes); the I-001 property test for every perturbation module (a known-correct solution still verifies on the perturbed variant); verifier correctness against hand-written passing and failing transcripts per family; scaffold identity across adapters (identical request payloads modulo provider envelope, asserted on the smoke agent); cost-cap enforcement; resumability (kill and re-run completes a cell); results-schema checks; secret-hygiene test asserting no environment values appear in transcripts or logs.
• The implementation stage runs: ruff check, ruff format --check, mypy, pytest -q, and agent-bench generate --smoke && agent-bench perturb --smoke && agent-bench run --smoke && agent-bench score --smoke && agent-bench analyze --smoke as its startup smoke.
• Only checkable by hand: figure legibility, and a spot audit of ~20 episode transcripts via agent-bench view against their verifier verdicts.

Definition of done

• All Must requirements implemented; the smoke pipeline prints SMOKE OK on a clean checkout, offline, on CPU.
• The shipped experiments.yaml defines a grid of ≥2 configurable models × 3 task families × 2 difficulties × all perturbation axes at their stated levels, executable without code edits and within the $200 cap; the smoke grid has actually been executed end-to-end by the pipeline.
• analyze outputs: per-axis delta figures, the rank-stability table, the variance decomposition, the power-analysis table, and the hypothesis verdict table (R-012), all regenerable from results.parquet alone.
• HYPOTHESES.md and experiments.yaml committed; usage documentation for every subcommand; test suite green; repo publishable as-is (Constraints last bullet).

Open questions

• Which two-to-three models to name in the shipped manifest's default grid — decided at run time by whoever executes it (pricing entries make any OpenAI-compatible or Anthropic endpoint usable); the pipeline must not hardcode any.
• Whether the tabular verifier compares exact bytes or canonicalized CSV (column order, float formatting). Canonicalized is more forgiving and arguably fairer; exact is simpler and stricter. Plan stage decides once, identically for all cells, and documents it.
• Paraphrase templates: hand-written (self-contained, auditable) vs. LLM-generated then frozen (more natural, needs one-time generation and human review). Default to hand-written unless decided otherwise.
• Whether partial_score is meaningful for filesys goals (fraction of goal conditions met) or binary-only everywhere. Plan stage proposes per family; binary is the fallback.