Broken Campus 🏚️

A production-reliability benchmark for tool-using LLM agents. Every model aces the happy path — Broken Campus scores what an agent does when the knowledge base is missing the fact, a tool lies, a retrieved document is poisoned, or the output contract is under attack. Measured identically across providers (NVIDIA NIM / OpenAI / Anthropic) on one strict JSON contract and one machine-readable trace, graded by deterministic code — no LLM-as-judge.

Author: B Torkian, NVIDIA Developer Champion. Companion to the Build Production AI Agents with NVIDIA Nemotron series. See How to cite below.

Status: v0.1 — public, reproducible, MIT/CC-BY. Bring your own API keys.

---

Why you can trust the numbers (fairness + reproducibility)

This benchmark is built to survive a skeptic, because the author works with NVIDIA and any hidden bias would be disqualifying. Every design choice below was hardened after an independent adversarial fairness audit:

• Bring-your-own-keys. No keys ship in this repo (only .env.example). You run it against your own accounts and get your own numbers.
• Synthetic, post-cutoff data. The knowledge base uses invented (nonce) facts, so "the fact was deleted" genuinely means no model could know it — not "a frontier model happened to memorize a real fact." No personal or real-world data.
• Deterministic everything. Keyword retrieval, seeded fault injection, and pure-code assertions on the JSON contract + trace. No LLM judges grading other LLMs. A skeptic can re-derive every PASS/FAIL by hand from the published traces.
• Native APIs per model. Each model uses its own recommended structured-output / tool-calling interface and decoding settings — no model is handicapped by another's prompt style. Exact wire requests are published; results are labeled endpoint + adapter, not "pure model."
• Raw-first-response is primary. JSON is repaired deterministically (a library, applied equally); any repair is reported separately and counted in cost/latency.
• Matched clean controls. Every corrupted case has an identical clean twin; we report the attack-induced change, and put clean-task accuracy and false-refusal rate beside every safety metric so "always refuse" can never win.
• Preregistered + hashed protocol. Prompts, cases, checkers, and per-model settings are frozen and hashed before any scored run — run python hash_protocol.py to regenerate protocol.sha256 and confirm the frozen inputs are unchanged. Nothing is tuned while watching results.
• Everything is public. All cases, the KB, poison sets, tool schemas, checkers, the scorer, and per-model settings are in this repo, and a frozen scored run (scorecard + per-case records) is published under results/ (scorecard_multiseed.json, runs.jsonl, and the deep-dive records) so you can re-derive any verdict without spending a token. Confidence intervals are Wilson score intervals. Because the seeds are near-deterministic the effective sample is small (≈5 distinct cases per condition on the main suite), so treat the intervals as deliberately conservative and the fine-grained rankings as directional, not definitive — the deep-dive run raises the abstention sample to 40 distinct probes for exactly this reason.

Scope: a controlled micro-benchmark (small KB, few tools). Results are directional — which model to shortlist for a production agent — not a universal SLA. Claims are constrained to this agent + failure regime.

---

Install & run

git clone https://github.com/torkian/broken-campus
cd broken-campus
pip install -r requirements.txt
cp .env.example .env      # then paste your own NVIDIA / OpenAI / Anthropic keys
python list_models.py     # see the exact model IDs your keys can call
python smoke.py           # sanity check: all configured providers run one case
python run.py             # the full benchmark → results/ + scorecard   (see run.py --help)

You only need the keys for the providers you want to test; the rest are skipped.

Reproduce the published results

Every scored run writes results/manifest.json — the model ids, decoding settings, seed count, price-table version, and the protocol hash — next to the scorecard. The frozen launch run is committed under results/ (scorecard_multiseed.json, per-case runs.jsonl, manifest.json), so you can re-derive any PASS/FAIL from the traces, or re-run with the same settings and diff.

python hash_protocol.py            # verify the frozen protocol matches protocol.sha256
python run.py --help               # options: --models, --seeds
python run.py --models lightning   # try one model on a free NVIDIA key (fast subset)
python route_sim.py                # the routed "money table", offline from published records

---

License

• Code: MIT (see LICENSE).
• Data & cases (KB, poison sets, golden assertions): CC-BY-4.0 — reuse freely with attribution.

How to cite

If you use Broken Campus (the harness, the cases, or the methodology), please cite:

@software{torkian_broken_campus_2026,
  author  = {Torkian, B.},
  title   = {{Broken Campus}: A Production-Reliability Benchmark for Tool-Using Agents},
  year    = {2026},
  version = {0.1.0},
  url     = {https://github.com/torkian/broken-campus},
  note    = {NVIDIA Developer Champion}
}

A machine-readable CITATION.cff is included (GitHub shows a "Cite this repository" button). A versioned release will be archived to Zenodo for a citable DOI — add the DOI here once minted.