Repository navigation
What would you want proven about an agent's work? #7
surlymochan
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
OpenFlow is a CLI that checks one thing: whether a coding agent's "I wrote the test first" claim holds up. A red proof has to fail before the fix and a green proof has to pass after, both bound to a snapshot of the tree, so a proof recorded after the fact is rejected, and test changes whose assertions are empty or type-only are rejected too. It runs from Claude Code, Codex, Cursor, OpenCode, or a plain terminal.
The repository has the benchmark numbers, the scripts and the per-run logs, including the runs where it loses to a plain agent: https://github.com/surlymochan/openflow
I am looking for two things, and neither of them is a bug report.
Which part of an agent's work would you want proven, beyond the test? "The test failed before the change" is one narrow slice. The build was green before the change. The diff touched only the files it claimed. The agent read the file it says it read. The command it shows is the command it ran. Which of those would you want, and which would you still not trust with a proof attached?
If you have a setup where an agent lies about TDD, tell me the shape of it. The gates are narrow on purpose, so I would rather add the case that actually happens than the one I can imagine. Test written after the fix but before the claim, a test that passes against the old code, an assertion that only checks the type, a re-run that quietly reuses an earlier result.
I am the author, so please treat this as a request for the unfriendly version.
All reactions