Skip to content

v4.100.71

Choose a tag to compare

@mann1x mann1x released this 05 Sep 18:13
· 1411 commits to main since this release

The check the model is judged by is now a check it can run

The protocol's check reached the model from exactly one place — settle, at the
completion attempt — while propose_check told it the opposite: "it is run for
you, unattended, every time your turn ends, and its result comes back to you."

So a model that proposed a check believed it had a feedback loop, stopped
building one of its own, and got no verdict at all until the transaction was
over.

Measured on one workspace with nothing runnable in it, under 4.100.68: TX-01 ran
341 messages and 65 edits with nothing judged, then one SyntaxError: missing ) after argument list, then a full rollback — after which the model
copied the original file back over its work and started again. Across a dozen
runs there, the arm with a proposed check took four to six times the model
time of the self-declared verdict it replaced (18 minutes and a clean finish on
4.100.61; 41, 45, 81 and 107 minutes after), and closed nothing.

The arm that works has never had this problem, and not because its check is
better. There the check is a shell line named in the opening prompt, so the
model reruns it whenever it likes — 88 run_commands in one run that ended
FIXED. A check the model cannot type is a check it cannot consult.

run_check

No arguments, offered whenever the protocol is armed, including before any check
exists. It runs the current check against the working tree and settles
nothing and rolls nothing back
: the result is information, the transaction
stays open, the changes stay on disk. Ten per transaction, counted again on the
next one so a discarded attempt can still see what it died on. A check that
cannot run at all says so and says explicitly that this is not a reason to stop.

It is also the only way to reach a page check, which runs inside Cline and
cannot be typed at a shell — and it is offered for a configured oracle too,
where it saves the model retyping the command.

The description now says what happens

Which matters on its own: a model told it is being checked stops checking
itself. Both branches of the opening protocol prompt point at run_check as
well, with the same instruction — run it before you edit, so you see the
failure in the check's own words, and again after each change.

A switch, so the two can be compared

Settings → Features → Change Protocol → "Model proposes the check". On by
default. Off returns the no-check case to the model's own account of its work,
including standing down in Auto exactly as it did before the feature existed —
there is a test on that last part, because it is the easy thing to get wrong.

Verified

New tests run the check twice across an edit and assert the file is left as the
model wrote it and no transaction was settled; others cover the no-check
message, the per-transaction budget and its reset, and a check that throws.
Suites: core 3177, vscode 1516, llms 949, CLI 1226, agents 134, hub 111.