v0.8.0
Added
-
examples/invoice-push/— the fourth committed example and the
agent_looparchetype, completing node-kind coverage across the
examples (adapted from a real production browser-automation skill,
fully fictionalized): a bounded per-row browser cycle (5-node
loop_body, termination cap) that cannot crystallize into code,
surrounded by 12 deterministic nodes. First example to commit its
eval.yamlsidecar — with per-rowiterations, the calibration
fixture for the loop-aware cost model below. -
eval --runauto-wires the pipeline's MCP servers into the skill
trial. The compiled pipeline'smcp:bindings name exactly the
servers the source skill uses, so the "before" measurement now runs
the agent over the same live tools (--mcp-config+
--strict-mcp-config, per-servermcp__<server>__*allowlists;
URLs resolve from the binding's expliciturl, else
ROTE_MCP_<SERVER>_URL— the adapters' rule). Verified live against
a realclaude -prun over a Streamable-HTTP MCP server. -
Reliability flags on measured skill runs. Each trial is checked
structurally (never by an LLM) before it may calibrate priors:
errored,hit_max_turns(truncated — its cost is a floor, not a
measurement),suspiciously_few_turns(fewer turns than the pipeline
has data pulls — the agent never did the work), and
missing_mcp_servers(the skill ran without its tools). Flagged runs
are excluded fromsuggested_priorsre-fits, listed in the measured
scorecard section, and recorded in the calibration corpus with their
flags. -
Loop-aware before-cost model, calibrated against a production
browser-automation skill whose two real runs measured 184 and 730
agent turns (~$8.6 and ~$32 on Sonnet) against a prior estimate of
$0.82–$2.75:- The eval sidecar's
StepEstimategainsiterations: {low, high}—
a step that repeats per row/page/item declares its per-iteration
turns and realistic repeat count, and the whole-run turn estimate
multiplies them. Loop-dominated skills were previously understated
5–20× because the schema could not express iteration at all (the
compiler's own notes described the loop economics correctly; the
form had nowhere to put the number). - New
transcript_cap_tokensprior (default 165k, the per-turn
cache-read plateau measured on both production runs): the transcript
an agent re-reads saturates at the harness's compaction ceiling
instead of growing without bound, so cached-read totals transition
from quadratic to linear on long runs. Without the cap, correcting
the turn count would have swung the error the other way (~2.5×
over). - The compiler rubric's calibration anchors are now regime-aware:
sequential tool-heavy skills anchor at 30–57 turns, per-item loop
skills at hundreds (the old universal 30–57 anchor actively pulled
loop-skill estimates down an order of magnitude).
Re-estimated with both fixes, the calibration skill's scorecard
brackets reality: $1.96–$22.68 estimated vs $8.59–$32.00 measured
(Sonnet), 98M estimated token ceiling vs 96M measured. - The eval sidecar's
Fixed
rote compilenow re-pointseval.yaml'ssource_skillalongside
pipeline.yaml's — the sidecar previously kept the agent's
temp-work-dir-relative path, a dead pointer in every kept compilation.