12,000 green unit tests, and the bugs still walked past all of them #213357
Replies: 1 comment
|
Thank you for your interest in contributing to our community! We currently only accept discussions created through the GitHub UI using our provided discussion templates. Please re-submit your discussion by navigating to the appropriate category and using the template provided. This discussion has been closed because it was not submitted through the expected format. If you believe this was a mistake, please reach out to the maintainers. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
A write-up from a team that ships entirely agent-written code has a number in it that I haven't been able to stop thinking about. Their end-to-end suite had run around 2,000 times. Eighty of those runs caught real defects — not flaky noise, actual broken behaviour that had already passed through twelve thousand unit tests. Every one of those tests was green. Every one of them was written by the same agent that wrote the code.
That's the part people miss. The failure isn't "AI writes bad tests." It's that the test and the implementation come from the same reading of the requirement. If the agent misunderstood the spec, the test encodes the misunderstanding and the code satisfies it. Green stops meaning "correct" and starts meaning "self-consistent."
Once you know the fingerprints, they're everywhere:
toHaveBeenCalledWith, and nothing checks what was actually producedtry/catchwhere the catch block never proves it was reachedexpect(result).toBeDefined()as the entire validationgetUser(null), no expired token, no user who shouldn't have permissionI've written all five of those myself, in a hurry, without any model involved. The difference is that a human's shallow test feels shaky when you read it. The generated one looks like it came from a senior engineer, which is exactly what makes it dangerous.
So I'm curious what actually holds the line in your repo. Mutation testing? Deleting the implementation and watching the test go red? Seeded bugs? A human who owns the expected values and refuses to let the model pick them?
I'm less interested in tool names than in which of these survived contact with a real team and didn't quietly become another red X that everyone reruns until it passes.
All reactions