flow v0.21.0 — eval-trust hardening + express-lane KILLED by data

Two-part release, both evidence-driven from the first REAL gate-eval baseline.

Motivating incident (260710)

The very first real batch after v0.19 shipped came back 17/18 INVALID despite the existing in-run retry — the storm produced no usable votes and burned all 18 billable calls. The only diagnostic signal (SessionEnd hook cancelled ×18 on stderr) was thrown away because the seam captured stdout only. A single call + one full rerun immediately after came back clean (hollow-flag 3/3 stages 100%, 0 INVALID/18) — mechanism unconfirmed at the time. The next storm has to be diagnosable, and the aborted-batch cost has to be capped. That's the whole shape of Phase 1.

Express-lane KILL (roadmap A → rejected by data)

Same-day per-cycle telemetry mining dissolved the original justification for an express-lane verb: cycles that ran ≥1 successful next reach Cards at 14/15 (93%), contract-stage dwell median is 40s (n=12; the "1.3h bottleneck" was a measurement artifact of usage --global averaging), and "33% abandonment" decomposes into exploration pokes + brownfield card-mode. Roadmap A is formally killed with a re-trigger condition. See docs/quality-metrics.md §A-kill.

Red-team pre-ship (3 hostile lenses, code-reviewer subagents, file:line-backed)

26 raw findings → 14 accepted after dedup (2 Critical, 5 High, 7 Medium). Catches include the original circuit-breaker missing its own motivating 17/18 incident (retrip on first-UNRELIABLE, not all-invalid) and the raw-capture spec being stdout-only when the storm signature was stderr — both would have shipped as latent bugs without the pass. Adjudication table lives in the plan.

Phase 1 — eval robustness

• Raw capture on final-INVALID — both attempts' stdout + stderr + rc persisted to .flow/eval-raw/<run_id>/<fixture>-v<vote>-a<attempt>.{out,err,rc}. Envelope stripped down to assistant/result/rate_limit_event records — cwd (which embeds the Windows username on this dev OS), session_id (resumable via claude --resume), plugin/memory paths, apiKeySource are all removed. cmd_eval calls _ignore_run_state so .flow/eval-raw/ is git-ignored.
• Circuit breaker on first-UNRELIABLE. Trip after the first fixture returns UNRELIABLE (invalid_count*3 > n), print an abort line naming the raw dir + file count, set an aborted flag, and skip the done trailer so --report/drift never surface the junk batch as canonical baseline. Distinct nonzero exit 2. --keep-going overrides (worst case ≈ 37 calls documented next to the flag).
• Injectable backoff. FLOW_EVAL_RETRY_BACKOFF env (default 5s, tests set 0). Retry now skipped when a rate-limit signal fired on attempt 1 OR when rc=124 (timeout) — both are infra failures, not formatting slips. A greppable retrying vote N line makes the retry path assertable via text, not stopwatch.
• Rate-limit visibility (advisory). _eval_parse_rate_limited anchored to rate_limit_info's own status value. Empirical: on cli 2.1.201 a healthy allowed event carries overageStatus":"rejected" as a separate field in the same envelope — a naive grep would false-positive on every healthy call. Documented best-effort until a real throttled sample lands.
• Pre-batch raw-dir prune. Keep the 3 newest run dirs by the epoch embedded in run_id (deterministic, mount-independent, unforgeable). TTL guard: never prune a dir whose embedded epoch is within FLOW_LOCK_TTL (900s) of now.
• Fixture-id sanitized for write paths. fid sanitized through the nonce charset before touching the filesystem — a hand-edited or FLOW_EVAL_MANIFEST-overridden manifest cannot traverse out of eval-raw/.
• Raw-write failure is LOUD, not the file's usual 2>/dev/null || true telemetry-sink pattern.

Phase 2 — fixture f01a repair

Complaint #3 rewritten (lines 38-41 only) as a coherent online quote with a synthetic thread-style link; no more "paraphrased-with-permission during a pilot household interview" framing that the judge was correctly FLAGging as laundered interview data.

Phase 3 — canonical baseline + docs + version

Canonical billable baseline (run …-1783743592-…): 6/6 MATCH, 0 unreliable, 0 invalid, 18/18 calls parsed. Per-stage hollow-flag-rate 1/1 sound-pass-rate 1/1 across 01-research, 02-scope, card. Judge claude-opus-4-7, CLI 2.1.201, gate_rules_sha 3672145322. Recorded in .flow/eval-results.jsonl; eval --report surfaces this as the drift-baseline.

CI status (post-release)

GREEN 3/3 OS on GitHub Actions run 29141602431 after 2 rounds of macOS-only fixes:

• Round 1 (82a67c0): retry skips on rc=124 timeout + prune bash-3.2 refactor
• Round 2 (17677b1): prune via tempfile + while read < $tmpf (redirect, not pipe) + test E threshold 20→45s for the documented macOS _run_with_timeout fallback DEBT lane

Post-ship independent audit

Separate code-reviewer subagent audit after ship: 0 runtime defects, 6 red-team applied fixes spot-checked (RT-C1, RT-C2, RT-H3, RT-H4, RT-H5, RT-H9 all verified with file:line), 1 docs drift caught and fixed same session (95672ef).

CI switch to Azure Pipelines

Right after the v0.21.0 line landed, GitHub Actions runs 29149122349 and 29149390414 failed 3/3 OS with steps=[] — runner never started — the recurring billing/quota block on private repos returned. .github/workflows/ci.yml retired in 2c24032; Azure Pipelines (free-tier private, 1 parallel job / 1,800 min/mo) already covers the same 3-OS matrix via azure-pipelines.yml.

Deferred, disclosed

• macOS _run_with_timeout fallback watchdog DEBT (DEBT.md) — needs real macOS access to diagnose
• A real throttled rate_limit_info sample — rate_limited field stays best-effort/advisory until one lands in the corpus

Files

14 changed, +1279 / -42 lines on the v0.21.0 line (d22d274 → 82a67c0 → acac540 → 17677b1 → 998ccac → 95672ef), plus post-release 2c24032 (CI retire) + aa6d4d3 (README update).