Repository navigation
What should AI Coding MPG Report No. 2 test? #2
Replies: 8 comments 5 replies
|
Comment 1: Planner–worker split vs a single model Setup: Claude Code with Opus 5.5 as an orchestrator that only plans. It writes a self-contained brief with acceptance criteria for each task, hands the implementation to a Haiku 5.5 subagent, then checks the result against the brief. This is the setup we run every day. Claim to check: an Opus planner with Haiku workers is cheaper per solved task than Sonnet 5.5 alone, counting the orchestrator’s tokens, and solves more than Haiku alone. Why it matters: Report No. 1 shows Haiku is about 6× cheaper but solves a little less. The open question is whether a good brief closes that gap, or whether the planner’s overhead eats the saving. Either answer would change how people set up agents. Comment 2: Repair by the same worker vs a fresh one Setup: when a subagent’s first attempt fails the tests, send the failure back to the same subagent instead of starting a new one with the failure in its brief. Claim to check: sending the repair back to the same worker costs less per solved task than a fresh worker, and solves at least as much. Why it matters: Report No. 1 found a fresh session costs 13% more because it rebuilds its cache. On a repair the fresh worker also loses what it learned about the code. We work this way, but we haven’t measured it. Comment 3: How often “done” is true Setup: any setup already in the report. No new runs are needed; this is one more field per run. Claim to check: cheaper models claim “done” when the hidden tests fail more often than stronger models do. Record, per setup, the share of runs where the agent reports success but the tests fail. Why it matters: in our work, a worker’s success report often fails our own checks. A cheap model is only cheap if you can trust its “done”, or else you pay to verify every result. This rate may differ more between models than the solve rate does. |
Report No. 2: design draft for commentsAs promised above, @pettykng: here is the draft, built on your three suggestions and on what Report No. 1 showed when split by task difficulty. Comments on any part are welcome, from anyone. Status: draft for comments, 2026-11-14. Nothing here is frozen. The pre-registration is written and frozen only after comments, and before any paid run. What Report No. 1 left open
Questions1. Planner and worker (primary). Suggested by pettykng.
2. Repair by the same worker or a fresh one. Suggested by pettykng.
3. How often "done" is true. Suggested by pettykng.
Changes from Report No. 1
Kept from Report No. 1
CostNot yet estimated properly. Report No. 1 cost $121 for 984 runs. Four setups on a similar task set, plus the repair runs, should land in roughly the same range, depending on how many tokens the orchestrator uses. A small pilot will measure that before the main run, and the budget is fixed in the pre-registration. Open questions
|
Update to the Report No. 2 draft (2026-11-14)Three things changed after a closer look at Report No. 1's data. @pettykng, the second one affects your repair suggestion. 1. Task groups: description length and size, not file count.
The "hard" tasks were mostly the thinly described ones (median 7.5 words, against 93 for the easy ones), and Opus's lead came mostly from there. The number of files changed predicted little. So Report No. 2 will report every result for two groups fixed before any run: description under or over 15 words, and 30 or fewer changed lines of code or more. This is now on the Report No. 1 page too, with each task's numbers in It also sharpens the planner question: an Opus planner should help most where the description is thin, because it has to work out what is wanted and write a brief the worker can follow. 2. The repair test (same worker or a fresh one) moves to Report No. 3. 3. Prompts: suggestions until 2026-11-21. The design as it stands:
Still unchanged: nothing is frozen yet, and no paid run starts before the pre-registration is committed. |
|
Thanks for picking up all three — and the description-length split is a good catch. On groups: short commit messages often make a task underspecified rather than hard: the hidden tests check behaviour the message never asks for. Worth saying so in the report. And with ~14 tasks per cell in a 2×2 split, I'd pre-register description length as the primary grouping and lines changed as secondary, or most per-cell results will end up "not shown". On repair feedback — how we do it in practice: the planner writes acceptance tests from its own brief before the worker starts, and those stay fixed for the attempt. The worker can see and run them; a failure against them is the repair signal, and the repair goes back to the same worker with the failing output. Before accepting, the planner also tries a few cases the worker's tests didn't cover. So the visible signal comes from inside the setup, and the commit's hidden tests stay purely the judge. That could make the repair question runnable in Report No. 2 after all. It would also let you report how often planner tests pass while hidden tests fail, a natural companion to the false-"done" measure. On the orchestrator prompt: this is the setup we run daily (Comment 1). We'll post a cleaned-up version of our planner profile before the 21st. |
Planner prompt (orchestrator)Generic version of the orchestrator prompt we use every day in Claude Code, with our private project names, paths and tools removed. Adapted for an unattended study run: there is no owner to ask, so the planner settles open questions from the repository itself. You are the planner. You receive a task and a repository. You do not write the change yourself; you decide what done means, brief one worker to do it, and accept or reject the result against evidence. You may make a small fix yourself (see "Fixing what you find"). 1. Understand the task before you brief anyoneThe task text may be a single line. Before writing the brief, spend a few tool calls finding out what it means in this repository:
Do not read the whole repository. Stop once you can state the outcome and how it will be checked. 2. Write the acceptance tests firstBefore the worker starts, write the tests that decide whether the work is done, in the project's own test framework and location:
3. Brief the workerThe worker starts cold and sees only your brief. Use exactly these headings:
For the one or two riskiest failure rules, tell the worker to break the behavior once after it works, show the test failing, and restore it. 4. Check the result; do not trust "done"The worker reports in these fields: A green report proves only what its checks check. After the worker reports:
Scale this to the change: a one-line fix needs a quick look, a new feature needs the full pass. 5. Fixing what you find
6. FinishFinish only when the acceptance tests and the existing suite pass on the final revision and your own attack cases found nothing. In the final message, state plainly:
Never claim the task is solved on the worker's report alone. Worker promptGeneric version of the worker prompt we use, paired with the planner prompt. You implement one bounded change and prove it with checks. Your brief is everything you know; it names the outcome, the workspace, the decisions already made, the constraints, and the commands that decide when you are done.
Report in exactly these fields: |
Pilot 1: numbers (8 runs, $1.51)@pettykng, as promised. Records and a short note: https://github.com/kumarganduri/aimpg/tree/main/docs/study/report-2/pilot Two spare starlette commits (not in the main sample), each run once per setup:
1. The planner never handed work to the worker (0 of 2 runs). So this pilot could not confirm the worker's model, the test-file check, or what a run with a worker costs. As it ran, the setup was Opus alone plus its own tests, at $0.38 against $0.28. It's your prompt, so I'd rather you say how to fix it than change it myself. One option: the planner always hands the implementation to the worker, and the small-fix exception applies only after a worker has reported. If you post the wording you want, I'll run a second pilot with it, on one small and one larger spare commit. 2. Nothing was solved, by any setup. 3. False "done": 7 of 8 runs. 4. The harness. Budget. At these costs the four setups come to roughly $13, $3, $47 and at least $64 for 168 runs each, more on larger tasks. I'll set the cap after the second pilot. |
Uh oh!
There was an error while loading. Please reload this page.
Report No. 1 compared Claude Code on Haiku, Sonnet and Opus 5.5, Codex, and one token-saving tool on 56 real commits from 8 open-source repos (report).
Report No. 2 will test what you ask for. Reply with one suggestion per comment and 👍 the ones you want; the most-wanted go into the next pre-registration. A good suggestion names:
Ideas already raised: a blind quality review of the passing solutions (Report No. 1 only checked that tests pass), and harder, larger tasks.
Vendors are welcome to suggest their own product's setup. It's tested the same way, and you can review the setup before any run.
Want a result sooner? Run it on your own repo with
aimpg verifyand submit it to the scoreboard.All reactions