Attack Your Own Method

Attack your own method

Section 8 of the method. Test the method against real defects before you trust it.

My first version of this escalated to the expensive model only when the automated checks failed. I tested that against the twelve real defects the expensive reviewers had caught that week, and every one of them was, by definition, something the checks had not noticed. The rule would have called in the strong model zero times for the things that mattered.

That is why the critic runs on every change now, and why the holdout check was written. Run your own version of that test before you trust any of this, and note that attacking the method is what eventually found that the holdout check had never run at all.

Be honest about the ceiling while you are at it. Nine of those twelve defects would still pass the holdout check, because a check cannot see what a test does not test. It closes the other three, the tests that pass either way. It does not replace the critic.

---

Previous: Measure every batch | Home | Next: Half of what an agent makes is not text