Skip to content

v0.8.0 — eval/ops, self-review critic, dynamic handoffs

Choose a tag to compare

@mtecnic mtecnic released this 27 Jul 02:23
· 5 commits to main since this release

Three features across the harness/loop/graph layers, built to keep data / state / flow / logic cleanly separated. The engine gained exactly one additive, emit-only line.

Highlights

critic node — self-review that can't rubber-stamp

An independent judge scored against explicit criteria in a fresh context — it only ever sees the artifact + criteria, never the producing agent's conversation, so a model can't quietly grade its own work. LLM or deterministic handler: backend; the verdict routes via edges (loop to revise on fail, forward on pass), bounded by the producer's max_iterations.

Dynamic handoffs on agent

A handoffs: list adds one synthetic handoff_to_<target> tool per target; calling one transfers control (goto) and the full conversation (a state channel the target reads at <state.handoff.*>). Bounded by the existing step/iteration guards.

eval/ops — a read-only observer

A new graphx.eval package over the existing events + checkpoints tables — never touches the execution path, state, or routing.

  • Per-run + per-node reports: cost, latency, retries, fallbacks, dead-letters, interventions.
  • Dataset-driven eval: deterministic assertions + optional LLM judge (reuses the critic).
  • Version compare: same dataset vs two graphs → outcome / metric / trace diff.
  • Golden-trace normalization so behavioral regressions diff cleanly.
  • CLI: runs, run-show, eval, eval-compare. TUI run browser on b.

Verification

283 tests (20 new), ruff-clean, Python 3.12+. Live-verified against a local vLLM: critic review loop, handoff, eval run, and the TUI run browser.

Examples

examples/review_loop.yaml, examples/handoff.yaml, examples/hello.eval.yaml.

Full changelog: v0.7.1...v0.8.0