Repository navigation
v7.1.0
[7.1.0] - 2026-07-28
The last route to a dishonest completed closure is closed, and the two claims
that rested on prompt prose are now measured.
Qualified on Flow 7.0.2 / OpenCode 1.18.6 from the 81-run matrix of 2026-07-28T16:23Z
across opencode/claude-sonnet-5, openai/gpt-5.6-sol, and xai/grok-4.5: 81/81
passed, nothing excluded, nothing aborted, and all 81 cassettes reproduced without a
fidelity caveat. Two earlier matrices needed a merged re-run to qualify — one for a
wedged grep call, one for an attempt excluded over a question it was right to ask.
Neither recurred.
- A plan declares its canonical gate.
plan.gateis the exact command that
validates the whole repository, named at planning time and locked by approval. A
broadobservation has to run that command byte-for-byte, and the gate joins the
vetoed-command set, so its latest failure blocks review whatever scope an
observation claimed. This closes the escape ADR
0009 recorded as open: a measured
run had closedcompletedby claiminggit diff --checkas its gate — a command
that cannot fail, so nothing was ever observed red. - A plan declares evidence this host cannot produce.
plan.externalEvidence
names each acceptance observation needing an operating system, service, credential,
or device this machine may lack, with the exact command whose passing is that
observation. Final review andcompletedclosure are both refused until that
command passes, so a self-written proxy cannot stand in for it. The first
three-provider eval matrix found three runs substituting one — two closed
completedover a Windows-only criterion on a Linux host, with the independent
review passing, and the suite's own false-completion metric reported zero. Each
entry also names theplatformthat can observe it —win32,darwin,linux, or
otherfor a service, credential, or device — and Flow records the host every
observation ran on, so the declared command passing on the wrong machine no longer
discharges it. The next matrix run found exactly that: a declared Windows entry
cleared by its own command's exit zero on Linux, green because the Windows case is
skipped there. The refusal now distinguishes a command never observed from one that
passed on the wrong host, because those call for opposite moves. See
ADR 0011. - Acceptance evidence names test cases, not exit codes. Each
externalEvidence
entry also declaresassertions: the case names whose passing is that
observation.flow_validation_starttakesresultsPath, the JUnit XML the command
writes, and Flow records what that report said about each declared name. An entry is
satisfied only when every one is reportedpassed—skippedandabsent
discharge nothing. This closes the limitation ADR 0011 recorded rather than fixed:
comparing the host closed the wrong machine and left the same skip on the right
one, where a case guarded, filtered, renamed out of the run, or never written still
exits zero. The names come from the approved plan and never from the caller, and a
report that predates the arming is read as no report at all. Declare an empty list
when the evidence is not a test result; that keeps the exit-code rule, which is the
honest answer for a credential or a device. See
ADR 0012. - The reviewer can see the two commands the plan declares. Its plan context now
carriesgateandexternalEvidence, which it was asked to judge and was never
shown: it could not tell abroadobservation that ran the canonical gate from one
that ran something else, and could not check a declared environment against the host
the observation recorded. - Session v5 schema:
plan.gateis a new optional string. A document written by
an earlier build still hydrates and keeps the older rule wherebroadis the
claimant's word;flow_plan_saverequires the field for any new plan, so a plan
this build writes always declares one.plan.externalEvidenceis a new optional
array under the same rule: older documents declare nothing and owe no acceptance
observation, andflow_plan_saverequires the field — an empty list is the answer
when the goal is fully observable here. Each entry carries an optionalplatform
thatflow_plan_savelikewise requires for a new plan, and each validation
observation carries an optional runtime-writtenhostPlatform; an entry hydrated
without a platform keeps the command-only rule. Each entry also carries an optional
assertionslist thatflow_plan_saverequires for a new plan, and each observation
may carry aresultsPathand theobservedAssertionsthe runtime read from it; an
entry hydrated without assertions keeps the exit-code rule. Rolling an active session
back to a build without these fields is not supported, as with every previous
widening. - The bar is published, not described. What Flow
guarantees states which claims are enforced by types, attested
by the host, declared by the caller, judged by a model, or unenforced.
Release qualification publishes the eval
thresholds a release clears, andbun run qualifyapplies them. - Evals measure false completion and reviewer activity, derived from durable
documents rather than model prose, and a scheduled workflow runs the suite against
at least two providers weekly. A new scenario checks that a requirement no run can
observe is never reported as verified. A run that aborts mid-flight is counted and
excluded rather than scored as a failure — one wedged attempt was the only failing
threshold in a measured report, on a guarantee that never ran — andbun run qualifyrefuses a report holding one on a gated pair. A wedge diagnostic now names
the command each incomplete tool call was running. - Recorded model decisions replay for free. Every paid attempt now writes a
cassette — its tool calls, in order, with their arguments — andbun run replay
feeds those back through the real handlers against a fresh workspace with no model
and no host. It is deliberately the decision layer, not the HTTP wire: freezing tool
results too would mean Flow's own refusals never execute on replay, which is the one
class of defect this suite exists to catch. Every runtime change up to now needed
another paid matrix before anyone knew it had not broken a sequence a model already
performed. CI gates the committed cassettes; a run whose recording holds something a
decision-layer replay cannot reproduce is reported rather than gated. - A report can be read.
bun run triageranks a report's runs by how much
reading each is worth and prints its reason for each, because nobody should trust an
eval score without reading transcripts and there was no tooling for it: a 54-run
report was a table and a JSON file. It ranks rather than filters, and deliberately
does not flag a scored escalation every attempt of its pair made, or a lone silent
review pass — including both flagged 32 of 54 runs on a real report, almost all of
them the suite working. Excluding them flagged five: the wedge, the false completion,
and three escalation outliers. An empty result says so, since a suite that never
flags anything and one that measures nothing look identical from here. - Three eval tiers, three prices.
bun run replayis free and answers whether the
runtime still reaches the same outcome;bun run eval:smokeis one model and one
attempt, for a prompt change; the full matrix is the only thing that qualifies a
release. One price for every question is what made the suite something run at release
instead of during work. - A regression scenario for the hole named results closed.
skipped-case-refused
seeds a fixture whose Windows-only case is an ordinarytest.skipIf, so the declared
command runs here, on the right host, and exits zero. Declaring the command is no
longer enough; the plan has to name the case. It went 9/9 across three providers
twice and stays ungated anyway: every attempt declaredplatform: "win32"on a Linux host, so the platform rule refuses first and the named-case rule
is never what binds. What the scenario measures today is the declaration, not the
observation. unprovable-claim-refusedis gated at 90%. Measured 0/3, then 8/9, then 9/9 as
the rule and the prompts landed; the margin is one pair's own variance rather than an
allowance for refusals to fail. See
release qualification.continuation-acceptedis gated at 100%, and a new scenario plants a defect for
the reviewer. The first is the mirror ofgoal-change-refused: one rule is enforced
by refusing a changed goal and the other by accepting an unchanged one, so gating only
the refusal would let a build that refuses every continuation qualify. The second,
defect-fails-review, went 9/9 twice and never by the route it was built for — one
attempt left the planted defect in place, built past it, and the review passed without
mentioning it. It stays ungated because its rate measures the implementer, not the
reviewer. Silent review passes did fall below the ceiling for the first time, 38 of
42, so that metric can now move; the findings behind it were about untested edge cases
rather than the plant, which is the same limitation from the other side.- A question the next step answers no longer voids the attempt. An eval run that
stopped to ask the user was excluded wherever the scenario did not declare asking an
acceptable end — including a question asked partway through, which the following
step's prompt answers. Three scenarios open by planning, where asking for approval is
the behaviour another scenario gates at 100%, and one measured run lost its score for
doing exactly that. Since a gated pair needs three scored attempts, one correct
question could fail a release and force a paid re-run. Only a question the last step
ends on is excluded now. - Seven cassettes are pinned, so CI replays real decisions. One per scenario from
the 2026-07-28 matrix, spread across three providers. All 63 candidates from that run
replayed against this code; the only divergence was the attempt that wedged
mid-flight, which is advisory by construction. A run that ends by asking the user no
longer records a fidelity caveat either — the harness aborts that session itself, so
theMessageAbortedErrorit leaves behind was this suite's own doing, and calling it
unreproducible had made 19 of 63 cassettes advisory, every refusal scenario among
them. - Eval reports now include the reviewer. Subtask sessions are read too, so a
report finally contains the independent review's own tool calls. Before this, no
recorded report held a singleflow_feature_completecall, the check for rejected
submissions could never fire, and the reviewer's tokens went uncounted — so token and
cost totals from earlier reports are lower than the same runs would report now. /flow-autosays when your host cannot continue. Continuation needs assistant
message parentage; a host that does not report it now gets a plain note at startup
and anautoContinuationfield on status, instead of a lifecycle that appears to
stop after every feature for no reason.- Positioning, including when not to use Flow. See
positioning and the new README section.
The prompt surface got smaller, not larger: the typed gate replaced the prose that
asked the model to judge whether its own coverage claim was honest.
Install or update:
opencode plugin opencode-plugin-flow@7.1.0 --global --force