Month 3 Plan

Month 3 — Eval: The Month That Makes You an "AI Systems Engineer"

The plan's distinctive month and the heart of the project's unique claim. Being able to measure an agent system — to say "better" or "worse" with a number — is something very few people can do.

Learning core (both of you)

• Why an AI system must be tested; why LLM outputs are probabilistic
• Measuring agent success: did it complete, did it pick the right tool, in how many steps, at what cost
• Regression testing logic
• Eval set design
• LLM-as-a-judge: its power and its limits

Concepts you must be able to explain

• Eval set · [ ] Task success rate · [ ] Tool selection accuracy · [ ] Step efficiency
• Ground truth · [ ] LLM-as-a-judge · [ ] Regression · [ ] Non-determinism
• Eval pipeline · [ ] Baseline

Project steps

1. By hand, an eval set of 30–50 tasks (expected result, tools that should be selected, reasonable step count)
2. Task success measurement
3. Tool selection and step efficiency measurement
4. LLM-as-a-judge for answer quality (state the inconsistency risk honestly in the README)
5. A one-command measurement pipeline + summary report
6. Baseline recorded
7. Prove Month 2's system-prompt hypothesis with numbers

Why do we build the eval set by hand? Because the eval set itself is the "correct" definition of the system. If you measure with an unaudited ruler, all your improvements drift toward error. A small but audited set is far more valuable than a large but unaudited one.

Division of labor

Split the tasks and swap at the rotation — you decide who takes what. Two rough tracks: (a) metric implementation (success, tool selection, steps) + LLM-as-a-judge + baseline, (b) eval set writing + harness + reporting/CI integration.

It matters that the eval set is written by both people: a set written by one person carries that person's blind spots.

Content plan

• Manifesto video: "Everyone builds AI agents but no one measures whether they work well"
• Eval report screen recording
• A comparison clip showing the scores of two system prompts side by side
• "How to test an AI system" educational thread

Retro