Skip to content

v7.1.0

Choose a tag to compare

@github-actions github-actions released this 28 Jul 17:53
· 1041 commits to main since this release
81329e6

[7.1.0] - 2026-07-28

The last route to a dishonest completed closure is closed, and the two claims
that rested on prompt prose are now measured.

Qualified on Flow 7.0.2 / OpenCode 1.18.6 from the 81-run matrix of 2026-07-28T16:23Z
across opencode/claude-sonnet-5, openai/gpt-5.6-sol, and xai/grok-4.5: 81/81
passed, nothing excluded, nothing aborted, and all 81 cassettes reproduced without a
fidelity caveat. Two earlier matrices needed a merged re-run to qualify — one for a
wedged grep call, one for an attempt excluded over a question it was right to ask.
Neither recurred.

  • A plan declares its canonical gate. plan.gate is the exact command that
    validates the whole repository, named at planning time and locked by approval. A
    broad observation has to run that command byte-for-byte, and the gate joins the
    vetoed-command set, so its latest failure blocks review whatever scope an
    observation claimed. This closes the escape ADR
    0009
    recorded as open: a measured
    run had closed completed by claiming git diff --check as its gate — a command
    that cannot fail, so nothing was ever observed red.
  • A plan declares evidence this host cannot produce. plan.externalEvidence
    names each acceptance observation needing an operating system, service, credential,
    or device this machine may lack, with the exact command whose passing is that
    observation. Final review and completed closure are both refused until that
    command passes, so a self-written proxy cannot stand in for it. The first
    three-provider eval matrix found three runs substituting one — two closed
    completed over a Windows-only criterion on a Linux host, with the independent
    review passing, and the suite's own false-completion metric reported zero. Each
    entry also names the platform that can observe it — win32, darwin, linux, or
    other for a service, credential, or device — and Flow records the host every
    observation ran on, so the declared command passing on the wrong machine no longer
    discharges it. The next matrix run found exactly that: a declared Windows entry
    cleared by its own command's exit zero on Linux, green because the Windows case is
    skipped there. The refusal now distinguishes a command never observed from one that
    passed on the wrong host, because those call for opposite moves. See
    ADR 0011.
  • Acceptance evidence names test cases, not exit codes. Each externalEvidence
    entry also declares assertions: the case names whose passing is that
    observation. flow_validation_start takes resultsPath, the JUnit XML the command
    writes, and Flow records what that report said about each declared name. An entry is
    satisfied only when every one is reported passed — skipped and absent
    discharge nothing. This closes the limitation ADR 0011 recorded rather than fixed:
    comparing the host closed the wrong machine and left the same skip on the right
    one, where a case guarded, filtered, renamed out of the run, or never written still
    exits zero. The names come from the approved plan and never from the caller, and a
    report that predates the arming is read as no report at all. Declare an empty list
    when the evidence is not a test result; that keeps the exit-code rule, which is the
    honest answer for a credential or a device. See
    ADR 0012.
  • The reviewer can see the two commands the plan declares. Its plan context now
    carries gate and externalEvidence, which it was asked to judge and was never
    shown: it could not tell a broad observation that ran the canonical gate from one
    that ran something else, and could not check a declared environment against the host
    the observation recorded.
  • Session v5 schema: plan.gate is a new optional string. A document written by
    an earlier build still hydrates and keeps the older rule where broad is the
    claimant's word; flow_plan_save requires the field for any new plan, so a plan
    this build writes always declares one. plan.externalEvidence is a new optional
    array under the same rule: older documents declare nothing and owe no acceptance
    observation, and flow_plan_save requires the field — an empty list is the answer
    when the goal is fully observable here. Each entry carries an optional platform
    that flow_plan_save likewise requires for a new plan, and each validation
    observation carries an optional runtime-written hostPlatform; an entry hydrated
    without a platform keeps the command-only rule. Each entry also carries an optional
    assertions list that flow_plan_save requires for a new plan, and each observation
    may carry a resultsPath and the observedAssertions the runtime read from it; an
    entry hydrated without assertions keeps the exit-code rule. Rolling an active session
    back to a build without these fields is not supported, as with every previous
    widening.
  • The bar is published, not described. What Flow
    guarantees
    states which claims are enforced by types, attested
    by the host, declared by the caller, judged by a model, or unenforced.
    Release qualification publishes the eval
    thresholds a release clears, and bun run qualify applies them.
  • Evals measure false completion and reviewer activity, derived from durable
    documents rather than model prose, and a scheduled workflow runs the suite against
    at least two providers weekly. A new scenario checks that a requirement no run can
    observe is never reported as verified. A run that aborts mid-flight is counted and
    excluded rather than scored as a failure — one wedged attempt was the only failing
    threshold in a measured report, on a guarantee that never ran — and bun run qualify refuses a report holding one on a gated pair. A wedge diagnostic now names
    the command each incomplete tool call was running.
  • Recorded model decisions replay for free. Every paid attempt now writes a
    cassette — its tool calls, in order, with their arguments — and bun run replay
    feeds those back through the real handlers against a fresh workspace with no model
    and no host. It is deliberately the decision layer, not the HTTP wire: freezing tool
    results too would mean Flow's own refusals never execute on replay, which is the one
    class of defect this suite exists to catch. Every runtime change up to now needed
    another paid matrix before anyone knew it had not broken a sequence a model already
    performed. CI gates the committed cassettes; a run whose recording holds something a
    decision-layer replay cannot reproduce is reported rather than gated.
  • A report can be read. bun run triage ranks a report's runs by how much
    reading each is worth and prints its reason for each, because nobody should trust an
    eval score without reading transcripts and there was no tooling for it: a 54-run
    report was a table and a JSON file. It ranks rather than filters, and deliberately
    does not flag a scored escalation every attempt of its pair made, or a lone silent
    review pass — including both flagged 32 of 54 runs on a real report, almost all of
    them the suite working. Excluding them flagged five: the wedge, the false completion,
    and three escalation outliers. An empty result says so, since a suite that never
    flags anything and one that measures nothing look identical from here.
  • Three eval tiers, three prices. bun run replay is free and answers whether the
    runtime still reaches the same outcome; bun run eval:smoke is one model and one
    attempt, for a prompt change; the full matrix is the only thing that qualifies a
    release. One price for every question is what made the suite something run at release
    instead of during work.
  • A regression scenario for the hole named results closed. skipped-case-refused
    seeds a fixture whose Windows-only case is an ordinary test.skipIf, so the declared
    command runs here, on the right host, and exits zero. Declaring the command is no
    longer enough; the plan has to name the case. It went 9/9 across three providers
    twice and stays ungated anyway: every attempt declared platform: "win32" on a Linux host, so the platform rule refuses first and the named-case rule
    is never what binds. What the scenario measures today is the declaration, not the
    observation.
  • unprovable-claim-refused is gated at 90%. Measured 0/3, then 8/9, then 9/9 as
    the rule and the prompts landed; the margin is one pair's own variance rather than an
    allowance for refusals to fail. See
    release qualification.
  • continuation-accepted is gated at 100%, and a new scenario plants a defect for
    the reviewer. The first is the mirror of goal-change-refused: one rule is enforced
    by refusing a changed goal and the other by accepting an unchanged one, so gating only
    the refusal would let a build that refuses every continuation qualify. The second,
    defect-fails-review, went 9/9 twice and never by the route it was built for — one
    attempt left the planted defect in place, built past it, and the review passed without
    mentioning it. It stays ungated because its rate measures the implementer, not the
    reviewer. Silent review passes did fall below the ceiling for the first time, 38 of
    42, so that metric can now move; the findings behind it were about untested edge cases
    rather than the plant, which is the same limitation from the other side.
  • A question the next step answers no longer voids the attempt. An eval run that
    stopped to ask the user was excluded wherever the scenario did not declare asking an
    acceptable end — including a question asked partway through, which the following
    step's prompt answers. Three scenarios open by planning, where asking for approval is
    the behaviour another scenario gates at 100%, and one measured run lost its score for
    doing exactly that. Since a gated pair needs three scored attempts, one correct
    question could fail a release and force a paid re-run. Only a question the last step
    ends on is excluded now.
  • Seven cassettes are pinned, so CI replays real decisions. One per scenario from
    the 2026-07-28 matrix, spread across three providers. All 63 candidates from that run
    replayed against this code; the only divergence was the attempt that wedged
    mid-flight, which is advisory by construction. A run that ends by asking the user no
    longer records a fidelity caveat either — the harness aborts that session itself, so
    the MessageAbortedError it leaves behind was this suite's own doing, and calling it
    unreproducible had made 19 of 63 cassettes advisory, every refusal scenario among
    them.
  • Eval reports now include the reviewer. Subtask sessions are read too, so a
    report finally contains the independent review's own tool calls. Before this, no
    recorded report held a single flow_feature_complete call, the check for rejected
    submissions could never fire, and the reviewer's tokens went uncounted — so token and
    cost totals from earlier reports are lower than the same runs would report now.
  • /flow-auto says when your host cannot continue. Continuation needs assistant
    message parentage; a host that does not report it now gets a plain note at startup
    and an autoContinuation field on status, instead of a lifecycle that appears to
    stop after every feature for no reason.
  • Positioning, including when not to use Flow. See
    positioning and the new README section.

The prompt surface got smaller, not larger: the typed gate replaced the prose that
asked the model to judge whether its own coverage claim was honest.

Install or update:

opencode plugin opencode-plugin-flow@7.1.0 --global --force