Ran mutation testing (muteval) on experiment 0016

[AshwinUgale]

testing-agents.md lists mutation testing as one of the approaches for checking whether an eval suite would catch a regression, so after the significance layer (0026) I tried it on a eval in one your experiments. I maintain a tool for it — muteval — so that's what I used. I'm not here to pitch it; the idea is already in your doc, I just used the tool I had to run it.

Experiment 0016 (the promptfoo PR-scope classifier) was a convenient fixture — small, public, already here. Not picking on it, just needed a real eval to point at. muteval degrades the prompt (weakening modals, dropping/reordering/paraphrasing instructions) and reruns 0016's 8 cases against each mutated version, checking whether any case fails.

The result was what you'd expect: a fair amount of the prompt can be changed without any case failing, since the clear-cut cases pass on the model's priors either way. That's normal for a small golden set — it's just the kind of thing mutation testing makes visible.

A couple of caveats: I ran it on gpt-4o-mini rather than 0016's Vertex Sonnet (the promptfoo path wants an OpenAI-compatible endpoint), so the broad read holds across models but individual results don't. And on fullsend's real agentic evals this would only make sense as a periodic / release-cadence check, not a per-PR gate, since each mutant reruns the whole agent.

Sharing it here and asking where, if anywhere, it's worth taking. I'm agnostic between:

• writing it up as a short experiment here, using muteval;
• a fullsend-native version if you'd rather not depend on an outside tool — it'd overlap with muteval, so I wanted to put both on the table rather than assume;
• or leaving it as a reference here.

Happy to follow your lead. @rh-hemartin