v2 — Evals

Every capability after this one ships with evidence it works, instead of an assurance that it does.

v1 gave Cameron tools that touch your money. This release builds the layer that proves they behave: a unit suite over the payloads tools return, and an eval harness that drives the real agent against a sandbox Postgres, then grades what it actually did.

Built in public alongside the Agentailor blog — one tagged release per article. The series hub, including where the roadmap goes next, lives at blog.agentailor.com/cameron.

📖 The reasoning behind this release is written up in two articles:

• Testing Tools for AI Agents — the audit of Cameron's eight tools, the contract bugs it found, and the cheap test layer that catches them.
• How to Write AI Agent Evals That Prove You Wrong — sourcing cases, spotting the ones that pass for the wrong reason, and when to overrule a judge.

What's new in v2

🧪 An eval harness that runs the real agent

pnpm eval drives the actual agent — real model, real tools — against an isolated sandbox stack (Postgres 5545, MinIO 9110), then grades the run with plain functions. Deterministic graders, no LLM judge: assertions target structural facts (which tool ran, what landed in the database), which works here because the answer is usually a number and a number has one spelling. Runs are pass@k with repeats opt-in per case, multi-turn cases replay on one thread_id via the checkpointer, and every expected figure derives from the seeding formula rather than a hardcoded constant. Never in CI — these are slow, paid and non-deterministic by design.

🎯 11 cases across 5 defect classes

Grouped by what can go wrong, not by feature. A case earns its place by guarding something a unit test provably cannot:

Guards	Cases
Tool mis-selection — run_sql vs query_transactions, both directions	3
A capped page reported as a complete total	1
The approval gate — no mutation without it; a denial writes nothing	2
Prompt contracts — no double-prompting; recovery from an unknown category	2
CSV import — ambiguous date format, accented headers, nothing written before confirming	3

🐛 A bug the evals found, fixed in the same release

run_sql returned SQL NULL for a category that doesn't exist — a payload byte-identical to a category that exists with zero spending, so the agent guessed. It now attaches an explicit note when a result is empty or an all-NULL aggregate row. Same defect class as v1's truncation bug: a payload the agent predictably misreads.

📊 Runs you can read after the terminal scrolls away

Every run writes eval/results/latest.html — a standalone page, data inlined, failures sorted to the top — plus latest.json and a timestamped copy. Per-run grader verdicts with failure reasons, the tool trajectory, anything the approval gate paused, and the final answer: enough to diagnose without paying to re-run.

✅ 81 unit tests, free and offline

pnpm test needs no model, no network, no database, no API keys. Tests live beside the code they test and assert on the JSON payload a tool returns — the same surface the evals grade against.

🔒 CI that gates

ci.yml gates every PR on pnpm test + tsc --noEmit; release.yml gates v* tags on the same. pnpm typecheck:eval covers the eval tree, which the root tsc misses.

---

Also in v2

📋 A /capabilities page — the tool list is no longer source-only. Real names, real descriptions, and an approval badge per tool, derived from the same arrays the agent registers, so a new tool appears with no UI edit.

🎨 Cameron's own look — the generic starter chrome is gone. A paper/ink palette with a single amber accent that marks the approval boundary and nothing else, plus purpose-built tool renderers: SQL blocks, result tables, an expense receipt, and field grids, with a JSON fallback for every MCP tool.

📖 A rewritten README — leads with what Cameron is and the screenshots that show it, instead of the starter's feature list.

Upgrading from v1

No breaking changes and no new migrations. To run the evals:

docker compose -f compose.eval.yaml up -d   # sandbox stack — once
cp .env.eval.example .env.eval              # then add your ANTHROPIC_API_KEY
pnpm eval                                   # or: pnpm eval approval -v

The eval stack is deliberately separate from your dev stack, and db.mts refuses to run against any database not named cameron_eval — evals TRUNCATE, and the dev database is one port away.
/pull/1

Full Changelog: v1...v2