Skip to content

v0.8.0

Choose a tag to compare

@trevhud trevhud released this 30 Jul 04:18
de88e5b

Added

  • examples/invoice-push/ — the fourth committed example and the
    agent_loop archetype, completing node-kind coverage across the
    examples (adapted from a real production browser-automation skill,
    fully fictionalized): a bounded per-row browser cycle (5-node
    loop_body, termination cap) that cannot crystallize into code,
    surrounded by 12 deterministic nodes. First example to commit its
    eval.yaml sidecar — with per-row iterations, the calibration
    fixture for the loop-aware cost model below.

  • eval --run auto-wires the pipeline's MCP servers into the skill
    trial.
    The compiled pipeline's mcp: bindings name exactly the
    servers the source skill uses, so the "before" measurement now runs
    the agent over the same live tools (--mcp-config +
    --strict-mcp-config, per-server mcp__<server>__* allowlists;
    URLs resolve from the binding's explicit url, else
    ROTE_MCP_<SERVER>_URL — the adapters' rule). Verified live against
    a real claude -p run over a Streamable-HTTP MCP server.

  • Reliability flags on measured skill runs. Each trial is checked
    structurally (never by an LLM) before it may calibrate priors:
    errored, hit_max_turns (truncated — its cost is a floor, not a
    measurement), suspiciously_few_turns (fewer turns than the pipeline
    has data pulls — the agent never did the work), and
    missing_mcp_servers (the skill ran without its tools). Flagged runs
    are excluded from suggested_priors re-fits, listed in the measured
    scorecard section, and recorded in the calibration corpus with their
    flags.

  • Loop-aware before-cost model, calibrated against a production
    browser-automation skill whose two real runs measured 184 and 730
    agent turns (~$8.6 and ~$32 on Sonnet) against a prior estimate of
    $0.82–$2.75:

    • The eval sidecar's StepEstimate gains iterations: {low, high}
      a step that repeats per row/page/item declares its per-iteration
      turns and realistic repeat count, and the whole-run turn estimate
      multiplies them. Loop-dominated skills were previously understated
      5–20× because the schema could not express iteration at all (the
      compiler's own notes described the loop economics correctly; the
      form had nowhere to put the number).
    • New transcript_cap_tokens prior (default 165k, the per-turn
      cache-read plateau measured on both production runs): the transcript
      an agent re-reads saturates at the harness's compaction ceiling
      instead of growing without bound, so cached-read totals transition
      from quadratic to linear on long runs. Without the cap, correcting
      the turn count would have swung the error the other way (~2.5×
      over).
    • The compiler rubric's calibration anchors are now regime-aware:
      sequential tool-heavy skills anchor at 30–57 turns, per-item loop
      skills at hundreds (the old universal 30–57 anchor actively pulled
      loop-skill estimates down an order of magnitude).

    Re-estimated with both fixes, the calibration skill's scorecard
    brackets reality: $1.96–$22.68 estimated vs $8.59–$32.00 measured
    (Sonnet), 98M estimated token ceiling vs 96M measured.

Fixed

  • rote compile now re-points eval.yaml's source_skill alongside
    pipeline.yaml's — the sidecar previously kept the agent's
    temp-work-dir-relative path, a dead pointer in every kept compilation.